[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:58:38 dist-test-slave-gl2g krb5kdc[7117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:58:38 dist-test-slave-gl2g krb5kdc[7117](info): set up 2 sockets
Jun 20 21:58:38 dist-test-slave-gl2g krb5kdc[7117](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 21:58:39.532202 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:58:39 dist-test-slave-gl2g krb5kdc[7117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:58:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:58:39.655125 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:39811
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:39811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:39.738171 7130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:39.738389 7130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:39.738413 7130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:39.739625 7130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:39.739670 7130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:39.739684 7130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:39.739698 7130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:39.739710 7130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:39.741278 7130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:39811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:39811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:39.741552 7130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:39.741801 7130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:58:39.747267 7130 server_base.cc:1034] running on GCE node
W20260620 21:58:39.747233 7135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:39.747233 7138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:39.747319 7136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:39.747839 7130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:39.748157 7130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:39.749323 7130 hybrid_clock.cc:648] HybridClock initialized: now 1781992719749294 us; error 38 us; skew 500 ppm
Jun 20 21:58:39 dist-test-slave-gl2g krb5kdc[7117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:39.752120 7130 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:58:39.752705 7130 webserver.cc:460] Webserver started at http://127.6.241.190:42875/ using document root <none> and password file <none>
I20260620 21:58:39.752969 7130 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:39.753041 7130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:39.753190 7130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:39.754364 7130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "8bd35ee7a2964b388a069d91130ee8ec"
format_stamp: "Formatted at 2026-06-20 21:58:39 on dist-test-slave-gl2g"
server_key: "7ace070509b99fcd02778ae4b77eb1e9"
server_key_iv: "df5ec25928f1dcdca9fbd472197349d6"
server_key_version: "encryptionkey@0"
I20260620 21:58:39.754772 7130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "8bd35ee7a2964b388a069d91130ee8ec"
format_stamp: "Formatted at 2026-06-20 21:58:39 on dist-test-slave-gl2g"
server_key: "7ace070509b99fcd02778ae4b77eb1e9"
server_key_iv: "df5ec25928f1dcdca9fbd472197349d6"
server_key_version: "encryptionkey@0"
I20260620 21:58:39.756186 7130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:58:39.756917 7145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:39.757083 7130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:39.757189 7130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "8bd35ee7a2964b388a069d91130ee8ec"
format_stamp: "Formatted at 2026-06-20 21:58:39 on dist-test-slave-gl2g"
server_key: "7ace070509b99fcd02778ae4b77eb1e9"
server_key_iv: "df5ec25928f1dcdca9fbd472197349d6"
server_key_version: "encryptionkey@0"
I20260620 21:58:39.757264 7130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:39.784471 7130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:39.787134 7130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:39.787312 7130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:39.791282 7130 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:39811
I20260620 21:58:39.791321 7197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:39811 every 8 connection(s)
I20260620 21:58:39.791873 7130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:39.792531 7198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:39.795320 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec: Bootstrap starting.
I20260620 21:58:39.795938 7198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:39.796277 7198 log.cc:824] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:39.796991 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec: No bootstrap required, opened a new log
I20260620 21:58:39.798691 7198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd35ee7a2964b388a069d91130ee8ec" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39811 } }
I20260620 21:58:39.798834 7198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:39.798857 7198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd35ee7a2964b388a069d91130ee8ec, State: Initialized, Role: FOLLOWER
I20260620 21:58:39.799058 7198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd35ee7a2964b388a069d91130ee8ec" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39811 } }
I20260620 21:58:39.799139 7198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:39.799189 7198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:39.799242 7198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:39.800207 7198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd35ee7a2964b388a069d91130ee8ec" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39811 } }
I20260620 21:58:39.800319 7198 leader_election.cc:302] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bd35ee7a2964b388a069d91130ee8ec; no voters:
I20260620 21:58:39.800511 7198 leader_election.cc:288] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:58:39.800607 7201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:39.800773 7201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [term 1 LEADER]: Becoming Leader. State: Replica: 8bd35ee7a2964b388a069d91130ee8ec, State: Running, Role: LEADER
I20260620 21:58:39.800781 7198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:39.800927 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 7130
I20260620 21:58:39.801025 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:58:39.800881 7201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd35ee7a2964b388a069d91130ee8ec" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39811 } }
I20260620 21:58:39.801541 7203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd35ee7a2964b388a069d91130ee8ec. Latest consensus state: current_term: 1 leader_uuid: "8bd35ee7a2964b388a069d91130ee8ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd35ee7a2964b388a069d91130ee8ec" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39811 } } }
I20260620 21:58:39.801558 7202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bd35ee7a2964b388a069d91130ee8ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd35ee7a2964b388a069d91130ee8ec" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39811 } } }
I20260620 21:58:39.801666 7203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:39.801676 7202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:39.801832 7110 external_mini_cluster.cc:1428] Setting key 50e42d2f2393b5e7285da0ce9d549bc3
I20260620 21:58:39.802002 7210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:39.802538 7210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:39.804035 7210 catalog_manager.cc:1347] Generated new cluster ID: c11a152e2338478eb09fc687b5976ec8
I20260620 21:58:39.804088 7210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 21:58:39 dist-test-slave-gl2g krb5kdc[7117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:39.817090 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:39.802794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.6.241.190:39811)
0620 21:58:39.803668 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:39.803677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:39.803712 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:58:39.805213 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:39.805219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:39.805273 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:39.805405 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:39.806955 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.807891 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.808992 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.809426 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.809571 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:39.813132 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:39.813136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:39.813137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:39.813148 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:39.815072 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:39.815557 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:39.815561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:39.815561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:39.815655 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:39.815922 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:39.815923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:39.815923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:39.816198 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:39.816360 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:39.816644 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 21:58:39.816788 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":80,"threads_started":1}
I20260620 21:58:39.818573 7210 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:58:39.819139 7210 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:39.826350 7210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bd35ee7a2964b388a069d91130ee8ec: Generated new TSK 0
I20260620 21:58:39.826644 7210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:58:39.857596 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:39811
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:39.942024 7224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:39.942229 7224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:39.942256 7224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:39.943606 7224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:39.943647 7224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:39.943675 7224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:58:39.945098 7224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:39811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:39.945289 7224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:39.945466 7224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:39.950387 7229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:39.950363 7232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:39.950356 7230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:39.950546 7224 server_base.cc:1034] running on GCE node
I20260620 21:58:39.950898 7224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:39.951144 7224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:39.952289 7224 hybrid_clock.cc:648] HybridClock initialized: now 1781992719952276 us; error 25 us; skew 500 ppm
Jun 20 21:58:39 dist-test-slave-gl2g krb5kdc[7117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:39.954921 7224 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:58:39.955410 7224 webserver.cc:460] Webserver started at http://127.6.241.129:41465/ using document root <none> and password file <none>
I20260620 21:58:39.955657 7224 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:39.955703 7224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:39.955852 7224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:39.956929 7224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "898aa585d2c447c3907d329690d97b1c"
format_stamp: "Formatted at 2026-06-20 21:58:39 on dist-test-slave-gl2g"
server_key: "59397a3a54a1d257947f47a4980a2467"
server_key_iv: "ea2a856b12ceaf84e8a2a13c0759c311"
server_key_version: "encryptionkey@0"
I20260620 21:58:39.957307 7224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "898aa585d2c447c3907d329690d97b1c"
format_stamp: "Formatted at 2026-06-20 21:58:39 on dist-test-slave-gl2g"
server_key: "59397a3a54a1d257947f47a4980a2467"
server_key_iv: "ea2a856b12ceaf84e8a2a13c0759c311"
server_key_version: "encryptionkey@0"
I20260620 21:58:39.958890 7224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 21:58:39.959622 7239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:39.959815 7224 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:39.959914 7224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "898aa585d2c447c3907d329690d97b1c"
format_stamp: "Formatted at 2026-06-20 21:58:39 on dist-test-slave-gl2g"
server_key: "59397a3a54a1d257947f47a4980a2467"
server_key_iv: "ea2a856b12ceaf84e8a2a13c0759c311"
server_key_version: "encryptionkey@0"
I20260620 21:58:39.960016 7224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:39.971728 7224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:39.974265 7224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:39.974438 7224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:39.974745 7224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:39.975068 7224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:39.975123 7224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:39.975167 7224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:39.975196 7224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:39.981228 7224 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:43599
I20260620 21:58:39.981271 7352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:43599 every 8 connection(s)
I20260620 21:58:39.981623 7224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:58:39.982649 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 7224
I20260620 21:58:39.982743 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:58:39.982892 7110 external_mini_cluster.cc:1428] Setting key 731350107e8bf87dbe556d8eb2200e4d
Jun 20 21:58:39 dist-test-slave-gl2g krb5kdc[7117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
I20260620 21:58:39.992525 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:39.982362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:41655 (local address 127.6.241.190:39811)
0620 21:58:39.982451 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:39.982454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:39.982691 (+ 237us) server_negotiation.cc:407] Connection header received
0620 21:58:39.983380 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:39.983382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:39.983396 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:39.983463 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:39.984508 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.985332 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.986863 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.987071 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.987130 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:39.989413 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:39.989415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:39.989415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:39.989423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:39.991112 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:39.991530 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:39.991531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:39.991532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:39.991598 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:39.991840 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:39.991841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:39.991841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:39.991974 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:39.992054 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:39.992338 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 21:58:39.992427 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:58:39.992750 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:39.982354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:39811 (local address 127.6.241.129:41655)
0620 21:58:39.982541 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:39.982587 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:39.983264 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:39.983562 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:39.983566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:39.984192 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:39.984395 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:39.984400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.985445 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.985446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:39.986777 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:39.986780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.987144 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.987145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:39.987316 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:39.987935 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:39.987944 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:39.989314 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:39.991207 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:39.991213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:39.991215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:39.991443 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:39.991666 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:39.991667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:39.991667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:39.991765 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:39.992058 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:39.992065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:39.992198 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 21:58:39.992304 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 21:58:39.992499 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260620 21:58:39.993219 7353 heartbeater.cc:347] Connected to a master server at 127.6.241.190:39811
I20260620 21:58:39.993336 7353 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:39.993541 7353 heartbeater.cc:511] Master 127.6.241.190:39811 requested a full tablet report, sending...
I20260620 21:58:39.994148 7162 ts_manager.cc:194] Registered new tserver with Master: 898aa585d2c447c3907d329690d97b1c (127.6.241.129:43599)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
I20260620 21:58:39.995095 7162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:41655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:58:40.010136 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:39811
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:40.093374 7358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:40.093647 7358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:40.093685 7358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:40.094903 7358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:40.094977 7358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:40.095023 7358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:58:40.096601 7358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:39811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:40.096858 7358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:40.097090 7358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:40.102272 7366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:40.102284 7358 server_base.cc:1034] running on GCE node
W20260620 21:58:40.102291 7363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:40.102401 7364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:40.102818 7358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:40.103112 7358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:40.104257 7358 hybrid_clock.cc:648] HybridClock initialized: now 1781992720104246 us; error 29 us; skew 500 ppm
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:40.106791 7358 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:58:40.107234 7358 webserver.cc:460] Webserver started at http://127.6.241.130:46207/ using document root <none> and password file <none>
I20260620 21:58:40.107457 7358 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:40.107503 7358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:40.107648 7358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:40.108732 7358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "0c97f505017d42cabe2187dbecd32186"
format_stamp: "Formatted at 2026-06-20 21:58:40 on dist-test-slave-gl2g"
server_key: "3e85a7d3e876cf460404dba64b68978c"
server_key_iv: "2dbb2b11892bc1fbe5cb8cacd6c4aa91"
server_key_version: "encryptionkey@0"
I20260620 21:58:40.109119 7358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "0c97f505017d42cabe2187dbecd32186"
format_stamp: "Formatted at 2026-06-20 21:58:40 on dist-test-slave-gl2g"
server_key: "3e85a7d3e876cf460404dba64b68978c"
server_key_iv: "2dbb2b11892bc1fbe5cb8cacd6c4aa91"
server_key_version: "encryptionkey@0"
I20260620 21:58:40.110530 7358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 21:58:40.111404 7373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.111610 7358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:40.111706 7358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "0c97f505017d42cabe2187dbecd32186"
format_stamp: "Formatted at 2026-06-20 21:58:40 on dist-test-slave-gl2g"
server_key: "3e85a7d3e876cf460404dba64b68978c"
server_key_iv: "2dbb2b11892bc1fbe5cb8cacd6c4aa91"
server_key_version: "encryptionkey@0"
I20260620 21:58:40.111784 7358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:40.129565 7358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:40.131673 7358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:40.131830 7358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:40.132087 7358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:40.132418 7358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:40.132453 7358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.132495 7358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:40.132524 7358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.138707 7358 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:45523
I20260620 21:58:40.138753 7486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:45523 every 8 connection(s)
I20260620 21:58:40.139088 7358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:58:40.145332 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 7358
I20260620 21:58:40.145437 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:58:40.145614 7110 external_mini_cluster.cc:1428] Setting key 14af8df9c25ce56c2e2ef18c6142bda6
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:40.149469 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.139810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:42765 (local address 127.6.241.190:39811)
0620 21:58:40.139882 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.139884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.140109 (+ 225us) server_negotiation.cc:407] Connection header received
0620 21:58:40.141070 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.141072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.141084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.141152 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:40.142135 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.142755 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.143737 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.143923 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.143983 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.146473 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:40.146474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:40.146475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:40.146481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:40.147978 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.148499 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.148499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.148500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.148549 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.148831 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.148831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.148832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.148934 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:40.148984 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.149328 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.149401 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 21:58:40.149788 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.139791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:39811 (local address 127.6.241.130:42765)
0620 21:58:40.139967 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:40.140011 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:40.140941 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:40.141212 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:40.141216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:40.141814 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:40.142018 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.142024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.142836 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.142837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.143662 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.143664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.143998 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.143999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.144100 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.144621 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:40.144630 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:40.146372 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:40.148060 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:40.148069 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:40.148072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:40.148420 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:40.148611 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:40.148611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:40.148612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:40.148764 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:40.148999 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:40.149006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:40.149188 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 21:58:40.149310 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:58:40.149499 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260620 21:58:40.150401 7487 heartbeater.cc:347] Connected to a master server at 127.6.241.190:39811
I20260620 21:58:40.150525 7487 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:40.150756 7487 heartbeater.cc:511] Master 127.6.241.190:39811 requested a full tablet report, sending...
I20260620 21:58:40.151194 7162 ts_manager.cc:194] Registered new tserver with Master: 0c97f505017d42cabe2187dbecd32186 (127.6.241.130:45523)
I20260620 21:58:40.151587 7162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:42765
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:58:40.172720 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:39811
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:40.255271 7492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:40.255553 7492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:40.255594 7492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:40.256790 7492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:40.256862 7492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:40.256908 7492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:58:40.258437 7492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:39811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:40.258733 7492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:40.258984 7492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:40.264583 7498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:40.264602 7497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:40.264658 7492 server_base.cc:1034] running on GCE node
W20260620 21:58:40.264583 7500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:40.265051 7492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:40.265302 7492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:40.266440 7492 hybrid_clock.cc:648] HybridClock initialized: now 1781992720266427 us; error 29 us; skew 500 ppm
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:40.269313 7492 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:58:40.269790 7492 webserver.cc:460] Webserver started at http://127.6.241.131:36667/ using document root <none> and password file <none>
I20260620 21:58:40.269995 7492 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:40.270037 7492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:40.270129 7492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:40.271438 7492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "a347724d550b470bb046a4571d0be268"
format_stamp: "Formatted at 2026-06-20 21:58:40 on dist-test-slave-gl2g"
server_key: "36aadf1525bd68ba141ec1a06c715919"
server_key_iv: "6313f06259548a443e5a92c8e8d1d535"
server_key_version: "encryptionkey@0"
I20260620 21:58:40.271797 7492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "a347724d550b470bb046a4571d0be268"
format_stamp: "Formatted at 2026-06-20 21:58:40 on dist-test-slave-gl2g"
server_key: "36aadf1525bd68ba141ec1a06c715919"
server_key_iv: "6313f06259548a443e5a92c8e8d1d535"
server_key_version: "encryptionkey@0"
I20260620 21:58:40.273106 7492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:58:40.273856 7507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.274029 7492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:40.274099 7492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "a347724d550b470bb046a4571d0be268"
format_stamp: "Formatted at 2026-06-20 21:58:40 on dist-test-slave-gl2g"
server_key: "36aadf1525bd68ba141ec1a06c715919"
server_key_iv: "6313f06259548a443e5a92c8e8d1d535"
server_key_version: "encryptionkey@0"
I20260620 21:58:40.274153 7492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:40.286396 7492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:40.288880 7492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:40.289002 7492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:40.289216 7492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:40.289527 7492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:40.289561 7492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.289585 7492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:40.289598 7492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.295908 7492 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:36647
I20260620 21:58:40.295961 7620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:36647 every 8 connection(s)
I20260620 21:58:40.296268 7492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:58:40.297748 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 7492
I20260620 21:58:40.297852 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:58:40.298041 7110 external_mini_cluster.cc:1428] Setting key 1c80f53f0f9742903e34eb8a465b7333
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:40.306073 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.296972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:38191 (local address 127.6.241.190:39811)
0620 21:58:40.297058 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.297061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.297269 (+ 208us) server_negotiation.cc:407] Connection header received
0620 21:58:40.297884 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.297886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.297899 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.297994 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:40.299248 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.299755 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.300856 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.301051 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.301140 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.303682 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:40.303683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:40.303683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:40.303690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:40.304890 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.305258 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.305259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.305259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.305307 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.305533 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.305534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.305534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.305655 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:40.305705 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.305941 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.306003 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:58:40.306245 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.296961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:39811 (local address 127.6.241.131:38191)
0620 21:58:40.297142 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:40.297181 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:40.297768 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:40.298078 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:40.298082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:40.298925 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:40.299139 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.299143 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.299844 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.299845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.300745 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.300747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.301149 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.301150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.301278 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.301874 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:40.301882 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:40.303598 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:40.304967 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:40.304973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:40.304974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:40.305193 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:40.305366 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:40.305367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:40.305367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:40.305480 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:40.305709 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:40.305712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:40.305828 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 21:58:40.305914 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 21:58:40.306035 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
I20260620 21:58:40.306869 7621 heartbeater.cc:347] Connected to a master server at 127.6.241.190:39811
I20260620 21:58:40.306977 7621 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:40.307161 7621 heartbeater.cc:511] Master 127.6.241.190:39811 requested a full tablet report, sending...
I20260620 21:58:40.307564 7162 ts_manager.cc:194] Registered new tserver with Master: a347724d550b470bb046a4571d0be268 (127.6.241.131:36647)
I20260620 21:58:40.307929 7162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:38191
I20260620 21:58:40.310165 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:58:40.333977 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.325249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.6.241.129:43599)
0620 21:58:40.325836 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.325841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.325852 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:58:40.325878 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.325880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.325894 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.325968 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:40.326664 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.327413 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.328223 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.328563 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.328631 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.331082 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:40.331090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:40.331091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:40.331098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:40.332822 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.333176 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.333180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.333180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.333259 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.333433 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.333433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.333434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.333574 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:40.333690 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.333745 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.333815 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
W20260620 21:58:40.334386 7267 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56802
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:58:40.356873 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.349391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.6.241.129:43599)
0620 21:58:40.349584 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.349586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.349676 (+ 90us) server_negotiation.cc:407] Connection header received
0620 21:58:40.349787 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.349788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.349807 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.349863 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:40.350459 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.351043 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.351794 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.352003 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.352088 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.354456 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:40.354458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:40.354458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:40.354465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:40.355977 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.356268 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.356269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.356269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.356319 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.356472 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.356473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.356473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.356559 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:40.356648 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.356715 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.356778 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260620 21:58:40.357391 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 7224
I20260620 21:58:40.362499 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 7358
I20260620 21:58:40.367882 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 7492
I20260620 21:58:40.372231 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 7130
2026-06-20T21:58:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1900 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7635](info): set up 2 sockets
Jun 20 21:58:40 dist-test-slave-gl2g krb5kdc[7635](info): commencing operation
krb5kdc: starting...
W20260620 21:58:41.406138 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:58:41 dist-test-slave-gl2g krb5kdc[7635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:58:41.538579 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33791
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:33791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:41.620455 7648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:41.620680 7648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:41.620704 7648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:41.621799 7648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:41.621836 7648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:41.621851 7648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:41.621865 7648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:41.621876 7648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:41.623301 7648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:33791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:41.623494 7648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:41.623667 7648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:41.628640 7654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:41.628656 7653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:41.628679 7656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:41.628686 7648 server_base.cc:1034] running on GCE node
I20260620 21:58:41.629083 7648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:41.629376 7648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:41.630566 7648 hybrid_clock.cc:648] HybridClock initialized: now 1781992721630550 us; error 44 us; skew 500 ppm
Jun 20 21:58:41 dist-test-slave-gl2g krb5kdc[7635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992721, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:41.633081 7648 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:58:41.633466 7648 webserver.cc:460] Webserver started at http://127.6.241.190:38603/ using document root <none> and password file <none>
I20260620 21:58:41.633704 7648 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:41.633752 7648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:41.633898 7648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:41.635047 7648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "45ee5c0305f1442ead30e8807a277380"
format_stamp: "Formatted at 2026-06-20 21:58:41 on dist-test-slave-gl2g"
server_key: "4e4a4aafe23003f36b15cd92b609fec4"
server_key_iv: "a202bab104872876e00ad19ceab880cf"
server_key_version: "encryptionkey@0"
I20260620 21:58:41.635434 7648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "45ee5c0305f1442ead30e8807a277380"
format_stamp: "Formatted at 2026-06-20 21:58:41 on dist-test-slave-gl2g"
server_key: "4e4a4aafe23003f36b15cd92b609fec4"
server_key_iv: "a202bab104872876e00ad19ceab880cf"
server_key_version: "encryptionkey@0"
I20260620 21:58:41.636760 7648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:58:41.637494 7663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:41.637702 7648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:41.637794 7648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "45ee5c0305f1442ead30e8807a277380"
format_stamp: "Formatted at 2026-06-20 21:58:41 on dist-test-slave-gl2g"
server_key: "4e4a4aafe23003f36b15cd92b609fec4"
server_key_iv: "a202bab104872876e00ad19ceab880cf"
server_key_version: "encryptionkey@0"
I20260620 21:58:41.637867 7648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:41.661693 7648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:41.663920 7648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:41.664072 7648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:41.668140 7648 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:33791
I20260620 21:58:41.668175 7715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:33791 every 8 connection(s)
I20260620 21:58:41.668524 7648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:41.669445 7716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:41.671864 7716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380: Bootstrap starting.
I20260620 21:58:41.672400 7716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:41.672639 7716 log.cc:824] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:41.673280 7716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380: No bootstrap required, opened a new log
I20260620 21:58:41.673487 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 7648
I20260620 21:58:41.673575 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:58:41.673763 7110 external_mini_cluster.cc:1428] Setting key 64606085c81a29d9413fe7b89c23d4ee
I20260620 21:58:41.675709 7716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ee5c0305f1442ead30e8807a277380" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33791 } }
I20260620 21:58:41.675840 7716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:41.675864 7716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ee5c0305f1442ead30e8807a277380, State: Initialized, Role: FOLLOWER
I20260620 21:58:41.676025 7716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ee5c0305f1442ead30e8807a277380" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33791 } }
I20260620 21:58:41.676105 7716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:41.676163 7716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:41.676204 7716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:41.676901 7716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ee5c0305f1442ead30e8807a277380" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33791 } }
I20260620 21:58:41.677022 7716 leader_election.cc:302] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45ee5c0305f1442ead30e8807a277380; no voters:
I20260620 21:58:41.677170 7716 leader_election.cc:288] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:58:41.677230 7721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:41.677373 7721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [term 1 LEADER]: Becoming Leader. State: Replica: 45ee5c0305f1442ead30e8807a277380, State: Running, Role: LEADER
I20260620 21:58:41.677399 7716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:41.677472 7721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ee5c0305f1442ead30e8807a277380" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33791 } }
I20260620 21:58:41.677760 7721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45ee5c0305f1442ead30e8807a277380. Latest consensus state: current_term: 1 leader_uuid: "45ee5c0305f1442ead30e8807a277380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ee5c0305f1442ead30e8807a277380" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33791 } } }
I20260620 21:58:41.677855 7721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:41.677770 7723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45ee5c0305f1442ead30e8807a277380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ee5c0305f1442ead30e8807a277380" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33791 } } }
I20260620 21:58:41.678946 7723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380 [sys.catalog]: This master's current role is: LEADER
W20260620 21:58:41.679209 7737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:58:41.679261 7737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:58:41.679744 7729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:41.679955 7729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 21:58:41 dist-test-slave-gl2g krb5kdc[7635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:41.681988 7729 catalog_manager.cc:1347] Generated new cluster ID: adf27b7bc26e4f0698227f36b0a0348f
I20260620 21:58:41.682037 7729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:58:41.685731 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:41.674305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.6.241.190:33791)
0620 21:58:41.674510 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:41.674520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:41.674555 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:58:41.675265 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:41.675275 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:41.675339 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:41.675520 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:41.676187 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:41.677896 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:41.678904 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:41.679529 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:41.679642 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:41.682406 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:41.682409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:41.682410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:41.682422 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:41.684169 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:41.684562 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:41.684565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:41.684565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:41.684655 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:41.684868 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:41.684869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:41.684869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:41.685117 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:41.685286 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:41.685405 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 21:58:41.685546 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260620 21:58:41.701632 7729 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:58:41.702317 7729 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:41.712378 7729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45ee5c0305f1442ead30e8807a277380: Generated new TSK 0
I20260620 21:58:41.712607 7729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:58:41.747891 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33791
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:41.848129 7742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:41.848347 7742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:41.848371 7742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:41.849488 7742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:41.849530 7742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:41.849557 7742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:58:41.851112 7742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:41.851310 7742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:41.851531 7742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:41.856559 7748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:41.856578 7747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:41.856559 7750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:41.856930 7742 server_base.cc:1034] running on GCE node
I20260620 21:58:41.857117 7742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:41.857322 7742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:41.858460 7742 hybrid_clock.cc:648] HybridClock initialized: now 1781992721858447 us; error 28 us; skew 500 ppm
Jun 20 21:58:41 dist-test-slave-gl2g krb5kdc[7635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992721, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:41.861080 7742 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:58:41.861575 7742 webserver.cc:460] Webserver started at http://127.6.241.129:45843/ using document root <none> and password file <none>
I20260620 21:58:41.861810 7742 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:41.861855 7742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:41.862031 7742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:41.863168 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "35dc51be4c834c4e97850a0df479ec03"
format_stamp: "Formatted at 2026-06-20 21:58:41 on dist-test-slave-gl2g"
server_key: "74a2910d037dc24c0c16521fccd85950"
server_key_iv: "e333d8c0b17d36bae1ba8b5853d37691"
server_key_version: "encryptionkey@0"
I20260620 21:58:41.863541 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "35dc51be4c834c4e97850a0df479ec03"
format_stamp: "Formatted at 2026-06-20 21:58:41 on dist-test-slave-gl2g"
server_key: "74a2910d037dc24c0c16521fccd85950"
server_key_iv: "e333d8c0b17d36bae1ba8b5853d37691"
server_key_version: "encryptionkey@0"
I20260620 21:58:41.864909 7742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:58:41.865777 7757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:41.866004 7742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:41.866086 7742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "35dc51be4c834c4e97850a0df479ec03"
format_stamp: "Formatted at 2026-06-20 21:58:41 on dist-test-slave-gl2g"
server_key: "74a2910d037dc24c0c16521fccd85950"
server_key_iv: "e333d8c0b17d36bae1ba8b5853d37691"
server_key_version: "encryptionkey@0"
I20260620 21:58:41.866185 7742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:41.880295 7742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:41.882468 7742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:41.882721 7742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:41.882974 7742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:41.883306 7742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:41.883339 7742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:41.883361 7742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:41.883373 7742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:41.889300 7742 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:33665
I20260620 21:58:41.889349 7870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:33665 every 8 connection(s)
I20260620 21:58:41.889768 7742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:58:41.893666 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 7742
I20260620 21:58:41.893766 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:58:41.893924 7110 external_mini_cluster.cc:1428] Setting key 5e88bb272957e866263c7835e6f2737a
Jun 20 21:58:41 dist-test-slave-gl2g krb5kdc[7635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992721, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:41.900439 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:41.890570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:50189 (local address 127.6.241.190:33791)
0620 21:58:41.890669 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:41.890671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:41.890905 (+ 234us) server_negotiation.cc:407] Connection header received
0620 21:58:41.891911 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:41.891913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:41.891924 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:41.891978 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:41.892619 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:41.893113 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:41.894308 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:41.894653 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:41.894747 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:41.897094 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:41.897096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:41.897096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:41.897104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:41.898892 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:41.899372 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:41.899374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:41.899374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:41.899444 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:41.899738 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:41.899739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:41.899739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:41.899863 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:41.899939 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:41.900239 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 21:58:41.900334 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:58:41.900638 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:41.890576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33791 (local address 127.6.241.129:50189)
0620 21:58:41.890772 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:41.890821 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:41.891798 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:41.892037 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:41.892041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:41.892309 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:41.892525 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:41.892531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:41.893200 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:41.893201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:41.894210 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:41.894213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:41.894742 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:41.894744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:41.894881 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:41.895535 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:41.895544 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:41.896991 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:41.899006 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:41.899014 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:41.899017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:41.899271 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:41.899551 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:41.899552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:41.899552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:41.899667 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:41.899941 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:41.899949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:41.900100 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 21:58:41.900219 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 21:58:41.900386 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260620 21:58:41.901396 7871 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33791
I20260620 21:58:41.901511 7871 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:41.901695 7871 heartbeater.cc:511] Master 127.6.241.190:33791 requested a full tablet report, sending...
I20260620 21:58:41.902293 7680 ts_manager.cc:194] Registered new tserver with Master: 35dc51be4c834c4e97850a0df479ec03 (127.6.241.129:33665)
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
I20260620 21:58:41.903007 7680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:50189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:58:41.921027 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33791
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:42.004410 7876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:42.004629 7876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:42.004654 7876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:42.005833 7876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:42.005878 7876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:42.005906 7876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:58:42.007458 7876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:42.007726 7876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:42.007985 7876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:42.013136 7884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:42.013145 7882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:42.013190 7881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:42.013347 7876 server_base.cc:1034] running on GCE node
I20260620 21:58:42.013569 7876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:42.013842 7876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:42.015008 7876 hybrid_clock.cc:648] HybridClock initialized: now 1781992722014998 us; error 31 us; skew 500 ppm
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[7635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992722, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:42.017695 7876 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:58:42.018107 7876 webserver.cc:460] Webserver started at http://127.6.241.130:40623/ using document root <none> and password file <none>
I20260620 21:58:42.018368 7876 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:42.018414 7876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:42.018549 7876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:42.019652 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "037cdcdb687445b997fadb6dec86d94c"
format_stamp: "Formatted at 2026-06-20 21:58:42 on dist-test-slave-gl2g"
server_key: "26cf45b1525c41358b8c01b831fad39f"
server_key_iv: "97b28ef162904123fab4a2c32b182546"
server_key_version: "encryptionkey@0"
I20260620 21:58:42.020008 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "037cdcdb687445b997fadb6dec86d94c"
format_stamp: "Formatted at 2026-06-20 21:58:42 on dist-test-slave-gl2g"
server_key: "26cf45b1525c41358b8c01b831fad39f"
server_key_iv: "97b28ef162904123fab4a2c32b182546"
server_key_version: "encryptionkey@0"
I20260620 21:58:42.021399 7876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:58:42.022192 7891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:42.022398 7876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:42.022473 7876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "037cdcdb687445b997fadb6dec86d94c"
format_stamp: "Formatted at 2026-06-20 21:58:42 on dist-test-slave-gl2g"
server_key: "26cf45b1525c41358b8c01b831fad39f"
server_key_iv: "97b28ef162904123fab4a2c32b182546"
server_key_version: "encryptionkey@0"
I20260620 21:58:42.022521 7876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:42.042409 7876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:42.044543 7876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:42.044703 7876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:42.044957 7876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:42.045302 7876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:42.045362 7876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:42.045406 7876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:42.045433 7876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:42.051209 7876 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:35613
I20260620 21:58:42.051251 8004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:35613 every 8 connection(s)
I20260620 21:58:42.051617 7876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:58:42.056238 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 7876
I20260620 21:58:42.056334 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:58:42.056520 7110 external_mini_cluster.cc:1428] Setting key 0ce56f9b78766b1fa1a62b921bd0f9b5
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[7635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992722, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:42.061861 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.052414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:36995 (local address 127.6.241.190:33791)
0620 21:58:42.052485 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.052488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.052752 (+ 264us) server_negotiation.cc:407] Connection header received
0620 21:58:42.053691 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.053693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.053702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.053757 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:42.054383 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.054979 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.056044 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.056354 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.056423 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.059002 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:42.059003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:42.059004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:42.059011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:42.060420 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.060863 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.060864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.060865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.060925 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.061181 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.061182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.061183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.061298 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:42.061374 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.061676 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.061767 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:58:42.062073 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.052401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33791 (local address 127.6.241.130:36995)
0620 21:58:42.052622 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:42.052659 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:42.053587 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:42.053818 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:42.053821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:42.054075 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:42.054281 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.054286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.055077 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.055078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.055929 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.055931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.056460 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.056461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.056574 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.057206 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:42.057219 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:42.058902 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:42.060520 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:42.060528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:42.060530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:42.060771 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:42.061001 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:42.061001 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:42.061002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:42.061116 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:42.061365 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:42.061372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:42.061521 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 21:58:42.061652 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 21:58:42.061819 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":133,"threads_started":1}
I20260620 21:58:42.062882 8005 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33791
I20260620 21:58:42.063010 8005 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:42.063230 8005 heartbeater.cc:511] Master 127.6.241.190:33791 requested a full tablet report, sending...
I20260620 21:58:42.063728 7680 ts_manager.cc:194] Registered new tserver with Master: 037cdcdb687445b997fadb6dec86d94c (127.6.241.130:35613)
I20260620 21:58:42.064276 7680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:36995
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:58:42.081795 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33791
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:42.167323 8010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:42.167532 8010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:42.167555 8010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:42.168735 8010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:42.168778 8010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:42.168805 8010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:58:42.170339 8010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:42.170543 8010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:42.170797 8010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:42.175961 8018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:42.176000 8010 server_base.cc:1034] running on GCE node
W20260620 21:58:42.175952 8016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:42.175953 8015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:42.176378 8010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:42.176632 8010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:42.177768 8010 hybrid_clock.cc:648] HybridClock initialized: now 1781992722177757 us; error 27 us; skew 500 ppm
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[7635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992722, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:42.180467 8010 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:58:42.180915 8010 webserver.cc:460] Webserver started at http://127.6.241.131:35843/ using document root <none> and password file <none>
I20260620 21:58:42.181161 8010 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:42.181209 8010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:42.181373 8010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:42.182483 8010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "ea51f69e35a8466d917bff34a581ff2b"
format_stamp: "Formatted at 2026-06-20 21:58:42 on dist-test-slave-gl2g"
server_key: "01d04ea0ef24c2702e0251182807df0b"
server_key_iv: "6ec40c887527c8f944b64b44a8c71496"
server_key_version: "encryptionkey@0"
I20260620 21:58:42.182936 8010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "ea51f69e35a8466d917bff34a581ff2b"
format_stamp: "Formatted at 2026-06-20 21:58:42 on dist-test-slave-gl2g"
server_key: "01d04ea0ef24c2702e0251182807df0b"
server_key_iv: "6ec40c887527c8f944b64b44a8c71496"
server_key_version: "encryptionkey@0"
I20260620 21:58:42.184268 8010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:42.184955 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:42.185137 8010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:42.185233 8010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "ea51f69e35a8466d917bff34a581ff2b"
format_stamp: "Formatted at 2026-06-20 21:58:42 on dist-test-slave-gl2g"
server_key: "01d04ea0ef24c2702e0251182807df0b"
server_key_iv: "6ec40c887527c8f944b64b44a8c71496"
server_key_version: "encryptionkey@0"
I20260620 21:58:42.185318 8010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:42.214555 8010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:42.216495 8010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:42.216668 8010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:42.216912 8010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:42.217240 8010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:42.217300 8010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:42.217347 8010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:42.217371 8010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:42.223273 8010 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:38935
I20260620 21:58:42.223321 8138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:38935 every 8 connection(s)
I20260620 21:58:42.223634 8010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:58:42.226886 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 8010
I20260620 21:58:42.226967 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:58:42.227133 7110 external_mini_cluster.cc:1428] Setting key 2bfa648ac50ee85a04287b32022df521
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[7635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992722, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:42.233584 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.224317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:47371 (local address 127.6.241.190:33791)
0620 21:58:42.224399 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.224402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.224585 (+ 183us) server_negotiation.cc:407] Connection header received
0620 21:58:42.225701 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.225702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.225714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.225760 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:42.226416 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.227094 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.228463 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.228679 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.228764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.230900 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:42.230901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:42.230902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:42.230910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:42.232153 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.232576 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.232578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.232578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.232651 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.232908 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.232909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.232909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.233030 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:42.233107 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.233403 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.233485 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 21:58:42.233790 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.224297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33791 (local address 127.6.241.131:47371)
0620 21:58:42.224459 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:42.224502 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:42.225589 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:42.225830 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:42.225839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:42.226092 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:42.226288 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.226292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.227189 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.227190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.228367 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.228370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.228771 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.228772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.228918 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.229373 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:42.229382 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:42.230808 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:42.232250 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:42.232257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:42.232259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:42.232486 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:42.232718 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:42.232719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:42.232719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:42.232837 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:42.233103 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:42.233109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:42.233266 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 21:58:42.233383 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 21:58:42.233552 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260620 21:58:42.234236 8139 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33791
I20260620 21:58:42.234315 8139 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:42.234534 8139 heartbeater.cc:511] Master 127.6.241.190:33791 requested a full tablet report, sending...
I20260620 21:58:42.234899 7680 ts_manager.cc:194] Registered new tserver with Master: ea51f69e35a8466d917bff34a581ff2b (127.6.241.131:38935)
I20260620 21:58:42.235284 7680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:47371
I20260620 21:58:42.239082 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:58:42.246377 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.240635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.6.241.190:33791)
0620 21:58:42.240678 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.240680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.240783 (+ 103us) server_negotiation.cc:407] Connection header received
0620 21:58:42.240823 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.240823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.240831 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.240883 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:42.241452 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.241940 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.242662 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.242834 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.242885 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.243992 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:42.243993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:42.243993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:42.243999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:42.245379 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.245726 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.245726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.245727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.245777 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.245940 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.245941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.245941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.246069 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:42.246145 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.246230 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.246287 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 21:58:42.248620 7680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:42.250249 7680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:42.256959 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.252867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.6.241.131:38935)
0620 21:58:42.253318 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.253322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.253331 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:58:42.253368 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.253370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.253385 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.253523 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:42.253612 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.254784 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.256034 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.256530 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.256620 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.256681 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.256835 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.256874 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":83,"threads_started":1}
I20260620 21:58:42.256953 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.252684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:38935 (local address 127.0.0.1:46732)
0620 21:58:42.252982 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:42.252993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:42.253041 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:42.253522 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:42.253523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:42.253524 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:42.253567 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.253568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.254884 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.254885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.255943 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.255944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.256620 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.256621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.256702 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.256714 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 21:58:42.256817 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 21:58:42.256875 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":4,"thread_start_us":44,"threads_started":1}
I20260620 21:58:42.257071 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.252867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35613 (local address 127.0.0.1:45518)
0620 21:58:42.253148 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:42.253166 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:42.253208 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:42.253438 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:42.253442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:42.253445 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:42.253524 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.253526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.254463 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.254464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.255805 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.255807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.256817 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.256818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.256891 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.256894 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:58:42.256953 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:58:42.256992 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260620 21:58:42.257212 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.252684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:33665 (local address 127.0.0.1:42922)
0620 21:58:42.252908 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:42.252952 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:42.253023 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:42.253442 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:42.253443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:42.253446 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:42.253677 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.253679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.255007 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.255008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.255951 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.255953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.256830 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.256831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.256886 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.256889 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:58:42.256920 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 21:58:42.256966 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260620 21:58:42.257310 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.253099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.6.241.130:35613)
0620 21:58:42.253263 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.253267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.253277 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:58:42.253321 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.253323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.253335 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.253426 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:42.253623 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.254356 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.255894 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.256649 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.256723 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.256781 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.257187 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.257239 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260620 21:58:42.257889 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.252795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.6.241.129:33665)
0620 21:58:42.253260 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.253266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.253277 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:58:42.253319 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.253322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.253339 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.253755 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:42.253856 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.254897 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.256043 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.256744 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.257009 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.257745 (+ 736us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.257773 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.257815 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":100,"threads_started":1}
I20260620 21:58:42.257774 8073 tablet_service.cc:1467] Processing CreateTablet for tablet 1320d54db5174b6ea733df0e47c41654 (DEFAULT_TABLE table=test-table [id=d7b1199c2ff2451ba3e2edc96f9f7bfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:42.258189 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1320d54db5174b6ea733df0e47c41654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:42.258144 7939 tablet_service.cc:1467] Processing CreateTablet for tablet 1320d54db5174b6ea733df0e47c41654 (DEFAULT_TABLE table=test-table [id=d7b1199c2ff2451ba3e2edc96f9f7bfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:42.258471 7939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1320d54db5174b6ea733df0e47c41654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:42.258682 7805 tablet_service.cc:1467] Processing CreateTablet for tablet 1320d54db5174b6ea733df0e47c41654 (DEFAULT_TABLE table=test-table [id=d7b1199c2ff2451ba3e2edc96f9f7bfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:42.258997 7805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1320d54db5174b6ea733df0e47c41654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:42.261219 8156 tablet_bootstrap.cc:492] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c: Bootstrap starting.
I20260620 21:58:42.261581 8157 tablet_bootstrap.cc:492] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b: Bootstrap starting.
I20260620 21:58:42.262022 8156 tablet_bootstrap.cc:654] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:42.262272 8157 tablet_bootstrap.cc:654] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:42.262434 8158 tablet_bootstrap.cc:492] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03: Bootstrap starting.
I20260620 21:58:42.262704 8156 log.cc:824] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:42.262871 8157 log.cc:824] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:42.263190 8158 tablet_bootstrap.cc:654] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:42.263536 8156 tablet_bootstrap.cc:492] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c: No bootstrap required, opened a new log
I20260620 21:58:42.263625 8156 ts_tablet_manager.cc:1397] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:58:42.263700 8157 tablet_bootstrap.cc:492] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b: No bootstrap required, opened a new log
I20260620 21:58:42.263748 8158 log.cc:824] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:42.263775 8157 ts_tablet_manager.cc:1397] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:58:42.265035 8158 tablet_bootstrap.cc:492] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03: No bootstrap required, opened a new log
I20260620 21:58:42.265107 8158 ts_tablet_manager.cc:1397] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:58:42.265506 8157 raft_consensus.cc:348] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } }
I20260620 21:58:42.265698 8157 raft_consensus.cc:374] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:42.265733 8157 raft_consensus.cc:729] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea51f69e35a8466d917bff34a581ff2b, State: Initialized, Role: FOLLOWER
I20260620 21:58:42.265882 8157 consensus_queue.cc:260] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } }
I20260620 21:58:42.266010 8156 raft_consensus.cc:348] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } }
I20260620 21:58:42.266184 8157 ts_tablet_manager.cc:1428] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 21:58:42.266203 8156 raft_consensus.cc:374] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:42.266256 8156 raft_consensus.cc:729] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037cdcdb687445b997fadb6dec86d94c, State: Initialized, Role: FOLLOWER
I20260620 21:58:42.266297 8139 heartbeater.cc:503] Master 127.6.241.190:33791 was elected leader, sending a full tablet report...
I20260620 21:58:42.266402 8156 consensus_queue.cc:260] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } }
I20260620 21:58:42.266777 8156 ts_tablet_manager.cc:1428] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 21:58:42.267087 8005 heartbeater.cc:503] Master 127.6.241.190:33791 was elected leader, sending a full tablet report...
I20260620 21:58:42.267138 8158 raft_consensus.cc:348] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } }
I20260620 21:58:42.267307 8158 raft_consensus.cc:374] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:42.267344 8158 raft_consensus.cc:729] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dc51be4c834c4e97850a0df479ec03, State: Initialized, Role: FOLLOWER
I20260620 21:58:42.267446 8158 consensus_queue.cc:260] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } }
I20260620 21:58:42.267729 7871 heartbeater.cc:503] Master 127.6.241.190:33791 was elected leader, sending a full tablet report...
I20260620 21:58:42.267730 8158 ts_tablet_manager.cc:1428] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260620 21:58:42.302199 8006 tablet.cc:2369] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:58:42.391206 7872 tablet.cc:2369] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:58:42.428712 8162 raft_consensus.cc:482] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:42.428870 8162 raft_consensus.cc:504] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } }
I20260620 21:58:42.429364 8162 leader_election.cc:288] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35dc51be4c834c4e97850a0df479ec03 (127.6.241.129:33665), 037cdcdb687445b997fadb6dec86d94c (127.6.241.130:35613)
I20260620 21:58:42.432567 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.429614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:33665 (local address 127.6.241.131:33485)
0620 21:58:42.429685 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:42.429699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:42.429750 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:42.430010 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:42.430011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:42.430012 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:42.430098 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.430119 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.430942 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.430943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.431739 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.431742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.432294 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.432295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.432374 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.432378 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:42.432429 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 21:58:42.432462 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 21:58:42.432583 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.429750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:33485 (local address 127.6.241.129:33665)
0620 21:58:42.429826 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.429829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.429846 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:58:42.429872 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.429873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.429882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.429992 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:42.430228 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.430861 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.431845 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.432214 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.432273 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.432281 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.432454 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.432501 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:58:42.432633 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.429879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:49421 (local address 127.6.241.130:35613)
0620 21:58:42.429941 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.429943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.430092 (+ 149us) server_negotiation.cc:407] Connection header received
0620 21:58:42.430123 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.430124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.430136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.430194 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:42.430363 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.430877 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.431869 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.432234 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.432328 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.432337 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.432508 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.432551 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:58:42.432682 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.429777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35613 (local address 127.6.241.131:49421)
0620 21:58:42.429985 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:42.429997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:42.430082 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:42.430214 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:42.430215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:42.430216 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:42.430285 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.430287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.430982 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.430982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.431749 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:42.431751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.432352 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.432353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:42.432410 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.432433 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:58:42.432491 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 21:58:42.432528 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":127,"threads_started":1}
I20260620 21:58:42.432940 7825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1320d54db5174b6ea733df0e47c41654" candidate_uuid: "ea51f69e35a8466d917bff34a581ff2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35dc51be4c834c4e97850a0df479ec03" is_pre_election: true
I20260620 21:58:42.432994 7959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1320d54db5174b6ea733df0e47c41654" candidate_uuid: "ea51f69e35a8466d917bff34a581ff2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037cdcdb687445b997fadb6dec86d94c" is_pre_election: true
I20260620 21:58:42.433105 7825 raft_consensus.cc:2454] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea51f69e35a8466d917bff34a581ff2b in term 0.
I20260620 21:58:42.433154 7959 raft_consensus.cc:2454] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea51f69e35a8466d917bff34a581ff2b in term 0.
I20260620 21:58:42.433466 8029 leader_election.cc:302] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037cdcdb687445b997fadb6dec86d94c, ea51f69e35a8466d917bff34a581ff2b; no voters:
I20260620 21:58:42.433717 8162 raft_consensus.cc:2781] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:42.433805 8162 raft_consensus.cc:482] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:42.433826 8162 raft_consensus.cc:3037] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:42.434684 8162 raft_consensus.cc:504] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } }
I20260620 21:58:42.434844 8162 leader_election.cc:288] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [CANDIDATE]: Term 1 election: Requested vote from peers 35dc51be4c834c4e97850a0df479ec03 (127.6.241.129:33665), 037cdcdb687445b997fadb6dec86d94c (127.6.241.130:35613)
I20260620 21:58:42.435076 7825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1320d54db5174b6ea733df0e47c41654" candidate_uuid: "ea51f69e35a8466d917bff34a581ff2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35dc51be4c834c4e97850a0df479ec03"
I20260620 21:58:42.435097 7959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1320d54db5174b6ea733df0e47c41654" candidate_uuid: "ea51f69e35a8466d917bff34a581ff2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037cdcdb687445b997fadb6dec86d94c"
I20260620 21:58:42.435166 7825 raft_consensus.cc:3037] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:42.435186 7959 raft_consensus.cc:3037] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:42.435902 7825 raft_consensus.cc:2454] T 1320d54db5174b6ea733df0e47c41654 P 35dc51be4c834c4e97850a0df479ec03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea51f69e35a8466d917bff34a581ff2b in term 1.
I20260620 21:58:42.436039 7959 raft_consensus.cc:2454] T 1320d54db5174b6ea733df0e47c41654 P 037cdcdb687445b997fadb6dec86d94c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea51f69e35a8466d917bff34a581ff2b in term 1.
I20260620 21:58:42.436098 8029 leader_election.cc:302] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35dc51be4c834c4e97850a0df479ec03, ea51f69e35a8466d917bff34a581ff2b; no voters:
I20260620 21:58:42.436228 8162 raft_consensus.cc:2781] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:42.436417 8162 raft_consensus.cc:686] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [term 1 LEADER]: Becoming Leader. State: Replica: ea51f69e35a8466d917bff34a581ff2b, State: Running, Role: LEADER
I20260620 21:58:42.436527 8162 consensus_queue.cc:237] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } }
I20260620 21:58:42.437413 7679 catalog_manager.cc:5495] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b reported cstate change: term changed from 0 to 1, leader changed from <none> to ea51f69e35a8466d917bff34a581ff2b (127.6.241.131). New cstate: current_term: 1 leader_uuid: "ea51f69e35a8466d917bff34a581ff2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dc51be4c834c4e97850a0df479ec03" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "037cdcdb687445b997fadb6dec86d94c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea51f69e35a8466d917bff34a581ff2b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38935 } health_report { overall_health: HEALTHY } } }
W20260620 21:58:42.474262 8140 tablet.cc:2369] T 1320d54db5174b6ea733df0e47c41654 P ea51f69e35a8466d917bff34a581ff2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[7635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:58:42.489799 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.481191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.6.241.129:33665)
0620 21:58:42.481246 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.481248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.481330 (+ 82us) server_negotiation.cc:407] Connection header received
0620 21:58:42.481364 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.481364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.481376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.481423 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:42.482497 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.483107 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.483937 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.484148 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.484224 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.486854 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:42.486867 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:42.486867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:42.486875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:42.488555 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.489007 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.489011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.489012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.489080 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.489281 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.489281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.489282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.489433 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:42.489541 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.489628 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.489698 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[7635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[7635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:58:42.514115 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.506294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.6.241.129:33665)
0620 21:58:42.506379 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.506382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.506470 (+ 88us) server_negotiation.cc:407] Connection header received
0620 21:58:42.506569 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.506570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.506582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.506670 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:42.507263 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.507813 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.508588 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.508800 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.508910 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.511444 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:42.511448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:42.511448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:42.511461 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:42.512826 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.513206 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.513208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.513208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.513269 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.513478 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.513479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.513480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.513596 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:42.513698 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.513900 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.513994 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[7635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[7635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:58:42.538061 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:42.530548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.6.241.129:33665)
0620 21:58:42.530692 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:42.530695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:42.530808 (+ 113us) server_negotiation.cc:407] Connection header received
0620 21:58:42.530919 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:42.530920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:42.530931 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:42.531021 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:42.531629 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.532132 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.532892 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:42.533095 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:42.533183 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:42.535554 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:42.535556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:42.535557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:42.535567 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:42.536834 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.537217 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.537218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.537219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.537278 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:42.537476 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:42.537477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:42.537478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:42.537591 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:42.537702 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:42.537849 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 21:58:42.537935 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260620 21:58:42.538591 7785 tablet_service.cc:2908] Rejecting scan request for tablet 1320d54db5174b6ea733df0e47c41654: Uninitialized: clean time has not yet been initialized
I20260620 21:58:42.539522 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 7742
I20260620 21:58:42.544770 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 7876
I20260620 21:58:42.549466 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 8010
I20260620 21:58:42.554067 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 7648
2026-06-20T21:58:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2183 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[8180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[8180](info): set up 2 sockets
Jun 20 21:58:42 dist-test-slave-gl2g krb5kdc[8180](info): commencing operation
krb5kdc: starting...
W20260620 21:58:43.602269 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:58:43 dist-test-slave-gl2g krb5kdc[8180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:58:43.722568 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:37363
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:37363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:43.805055 8193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:43.805274 8193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:43.805297 8193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:43.806494 8193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:43.806542 8193 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:58:43.806556 8193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:43.806569 8193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:43.806663 8193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:43.806689 8193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:43.808257 8193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:37363
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:37363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:43.808525 8193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:43.808774 8193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:43.813656 8201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:43.813647 8199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:43.813648 8198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:43.813905 8193 server_base.cc:1034] running on GCE node
I20260620 21:58:43.814227 8193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:43.814525 8193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:43.815748 8193 hybrid_clock.cc:648] HybridClock initialized: now 1781992723815759 us; error 54 us; skew 500 ppm
Jun 20 21:58:43 dist-test-slave-gl2g krb5kdc[8180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992723, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:43.821161 8193 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:58:43.821691 8193 webserver.cc:460] Webserver started at http://127.6.241.190:40757/ using document root <none> and password file <none>
I20260620 21:58:43.821939 8193 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:43.822009 8193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:43.822161 8193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:43.823310 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "f5715b077ccf4a97afc94cb8365dc75e"
format_stamp: "Formatted at 2026-06-20 21:58:43 on dist-test-slave-gl2g"
server_key: "ca25c29d8d4fa850db979f12c176cade"
server_key_iv: "5a3a742ab7f856704806c6daabd5e626"
server_key_version: "encryptionkey@0"
I20260620 21:58:43.823683 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "f5715b077ccf4a97afc94cb8365dc75e"
format_stamp: "Formatted at 2026-06-20 21:58:43 on dist-test-slave-gl2g"
server_key: "ca25c29d8d4fa850db979f12c176cade"
server_key_iv: "5a3a742ab7f856704806c6daabd5e626"
server_key_version: "encryptionkey@0"
I20260620 21:58:43.825003 8193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 21:58:43.825780 8208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:43.825969 8193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 21:58:43.826076 8193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f5715b077ccf4a97afc94cb8365dc75e"
format_stamp: "Formatted at 2026-06-20 21:58:43 on dist-test-slave-gl2g"
server_key: "ca25c29d8d4fa850db979f12c176cade"
server_key_iv: "5a3a742ab7f856704806c6daabd5e626"
server_key_version: "encryptionkey@0"
I20260620 21:58:43.826152 8193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:43.845266 8193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:43.847675 8193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:43.847846 8193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:43.852772 8193 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:37363
I20260620 21:58:43.852806 8270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:37363 every 8 connection(s)
I20260620 21:58:43.853159 8193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:43.853789 8271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:43.856221 8271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e: Bootstrap starting.
I20260620 21:58:43.856851 8271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:43.857158 8271 log.cc:824] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:43.857805 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 8193
I20260620 21:58:43.857880 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:58:43.857843 8271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e: No bootstrap required, opened a new log
I20260620 21:58:43.858050 7110 external_mini_cluster.cc:1428] Setting key e00fe8b7a765827af1bdb538eb5ce0f4
I20260620 21:58:43.860204 8271 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5715b077ccf4a97afc94cb8365dc75e" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 37363 } }
I20260620 21:58:43.860374 8271 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:43.860438 8271 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5715b077ccf4a97afc94cb8365dc75e, State: Initialized, Role: FOLLOWER
I20260620 21:58:43.860608 8271 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5715b077ccf4a97afc94cb8365dc75e" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 37363 } }
I20260620 21:58:43.860697 8271 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:43.860721 8271 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:43.860773 8271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:43.861316 8271 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5715b077ccf4a97afc94cb8365dc75e" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 37363 } }
I20260620 21:58:43.861472 8271 leader_election.cc:302] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5715b077ccf4a97afc94cb8365dc75e; no voters:
I20260620 21:58:43.861675 8271 leader_election.cc:288] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:58:43.861716 8276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:43.861900 8276 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [term 1 LEADER]: Becoming Leader. State: Replica: f5715b077ccf4a97afc94cb8365dc75e, State: Running, Role: LEADER
I20260620 21:58:43.861948 8271 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:43.862072 8276 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5715b077ccf4a97afc94cb8365dc75e" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 37363 } }
I20260620 21:58:43.862694 8277 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5715b077ccf4a97afc94cb8365dc75e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5715b077ccf4a97afc94cb8365dc75e" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 37363 } } }
I20260620 21:58:43.862864 8278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5715b077ccf4a97afc94cb8365dc75e. Latest consensus state: current_term: 1 leader_uuid: "f5715b077ccf4a97afc94cb8365dc75e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5715b077ccf4a97afc94cb8365dc75e" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 37363 } } }
I20260620 21:58:43.862942 8277 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:43.863173 8278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:43.863409 8285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:43.863965 8285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:43.865653 8285 catalog_manager.cc:1347] Generated new cluster ID: 28a0b06b3fb1428fba7466a48faf2396
I20260620 21:58:43.865703 8285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 21:58:43 dist-test-slave-gl2g krb5kdc[8180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:43.872421 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:43.858637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.6.241.190:37363)
0620 21:58:43.858817 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:43.858827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:43.858862 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:58:43.860207 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:43.860216 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:43.860274 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:43.860445 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:43.861196 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:43.863889 (+ 2693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:43.865027 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:43.865523 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:43.865690 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:43.868250 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:43.868254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:43.868255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:43.868275 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:43.870727 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:43.871128 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:43.871132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:43.871132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:43.871245 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:43.871464 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:43.871465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:43.871465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:43.871746 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:43.871966 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:43.872044 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 21:58:43.872203 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260620 21:58:43.879352 8285 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:58:43.880115 8285 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:43.898377 8285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e: Generated new TSK 0
I20260620 21:58:43.898633 8285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:58:43.927801 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:37363
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:44.028545 8297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:44.028760 8297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:44.028785 8297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:44.030119 8297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:44.030160 8297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:44.030181 8297 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:58:44.030236 8297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:58:44.031735 8297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.241.190:37363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:44.031986 8297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:44.032224 8297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:44.037402 8302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:44.037396 8305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:44.037402 8303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:44.037727 8297 server_base.cc:1034] running on GCE node
I20260620 21:58:44.037961 8297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:44.038216 8297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:44.039353 8297 hybrid_clock.cc:648] HybridClock initialized: now 1781992724039334 us; error 36 us; skew 500 ppm
Jun 20 21:58:44 dist-test-slave-gl2g krb5kdc[8180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:44.041941 8297 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:58:44.042344 8297 webserver.cc:460] Webserver started at http://127.6.241.129:36891/ using document root <none> and password file <none>
I20260620 21:58:44.042533 8297 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:44.042575 8297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:44.042737 8297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:44.043780 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "28ba9f732db143e1a0bee8c2fe477086"
format_stamp: "Formatted at 2026-06-20 21:58:44 on dist-test-slave-gl2g"
server_key: "2ae1e72d7e9444cf21de6bf96b9b05a3"
server_key_iv: "75562691a27d2f2116d13970f5a3a082"
server_key_version: "encryptionkey@0"
I20260620 21:58:44.044087 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "28ba9f732db143e1a0bee8c2fe477086"
format_stamp: "Formatted at 2026-06-20 21:58:44 on dist-test-slave-gl2g"
server_key: "2ae1e72d7e9444cf21de6bf96b9b05a3"
server_key_iv: "75562691a27d2f2116d13970f5a3a082"
server_key_version: "encryptionkey@0"
I20260620 21:58:44.045380 8297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:58:44.046101 8312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:44.046304 8297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:44.046411 8297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "28ba9f732db143e1a0bee8c2fe477086"
format_stamp: "Formatted at 2026-06-20 21:58:44 on dist-test-slave-gl2g"
server_key: "2ae1e72d7e9444cf21de6bf96b9b05a3"
server_key_iv: "75562691a27d2f2116d13970f5a3a082"
server_key_version: "encryptionkey@0"
I20260620 21:58:44.046523 8297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:44.061014 8297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:44.063776 8297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:44.063936 8297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:44.064509 8297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:44.064605 8297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:44.064643 8297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:44.064673 8297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:58:44 dist-test-slave-gl2g krb5kdc[8180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:44.071841 8297 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:39511
I20260620 21:58:44.071904 8427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:39511 every 8 connection(s)
I20260620 21:58:44.072224 8297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:58:44.073704 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 8297
I20260620 21:58:44.073782 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:58:44.073974 7110 external_mini_cluster.cc:1428] Setting key 00cbcd0754be6ee50bf441d341b12f89
I20260620 21:58:44.075032 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.065036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:35267 (local address 127.6.241.190:37363)
0620 21:58:44.065087 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.065093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.065377 (+ 284us) server_negotiation.cc:407] Connection header received
0620 21:58:44.065939 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.065940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.065953 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.066006 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:44.066926 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.067434 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.068549 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.068880 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.068965 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.071811 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:44.071813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:44.071814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:44.071829 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:44.073463 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.073967 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.073968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.073969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.074029 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.074318 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.074319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.074320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.074446 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:44.074522 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.074847 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.074940 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 21:58:44.075240 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.065057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:37363 (local address 127.6.241.129:35267)
0620 21:58:44.065252 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.065289 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.065825 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.066097 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.066100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.066497 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:44.066821 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.066829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.067534 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.067536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.068451 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.068454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.068986 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.068987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.069134 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.069691 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:44.069702 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:44.071716 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:44.073588 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.073596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.073598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.073891 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.074137 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.074138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.074138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.074245 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.074513 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:44.074519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:44.074713 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.074825 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.074990 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
I20260620 21:58:44.076084 8428 heartbeater.cc:347] Connected to a master server at 127.6.241.190:37363
I20260620 21:58:44.076213 8428 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:44.076414 8428 heartbeater.cc:511] Master 127.6.241.190:37363 requested a full tablet report, sending...
I20260620 21:58:44.077327 8225 ts_manager.cc:194] Registered new tserver with Master: 28ba9f732db143e1a0bee8c2fe477086 (127.6.241.129:39511)
I20260620 21:58:44.077919 8225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:35267
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
I20260620 21:58:44.083794 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.077189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:58807 (local address 127.6.241.190:37363)
0620 21:58:44.078168 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.078170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.078291 (+ 121us) server_negotiation.cc:407] Connection header received
0620 21:58:44.078373 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.078374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.078384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.078425 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:44.078818 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.079307 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.080024 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.080237 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.080315 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.081263 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:44.081264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:44.081265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:44.081271 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:44.082654 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.083008 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.083010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.083010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.083064 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.083266 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.083267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.083268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.083380 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:44.083456 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.083620 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.083697 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958}
I20260620 21:58:44.083808 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.077147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:37363 (local address 127.6.241.129:58807)
0620 21:58:44.078235 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.078246 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.078315 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.078438 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.078439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.078666 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:44.078743 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.078745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.079404 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.079405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.079936 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.079938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.080311 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.080312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.080387 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.080864 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:44.080870 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:44.081159 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:44.082754 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.082755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.082755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.082925 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.083136 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.083137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.083137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.083196 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.083447 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:44.083448 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:44.083502 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.083588 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.083698 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":60,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:58:44.099082 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:37363
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:44.184397 8437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:44.184630 8437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:44.184655 8437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:44.185794 8437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:44.185834 8437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:44.185854 8437 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:58:44.185873 8437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:58:44.187299 8437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.241.190:37363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:44.187511 8437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:44.187696 8437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:44.192837 8443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:44.192890 8437 server_base.cc:1034] running on GCE node
W20260620 21:58:44.192855 8445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:44.192837 8442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:44.193315 8437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:44.193576 8437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:44.194772 8437 hybrid_clock.cc:648] HybridClock initialized: now 1781992724194750 us; error 32 us; skew 500 ppm
Jun 20 21:58:44 dist-test-slave-gl2g krb5kdc[8180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:44.197275 8437 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:58:44.197799 8437 webserver.cc:460] Webserver started at http://127.6.241.130:42475/ using document root <none> and password file <none>
I20260620 21:58:44.198031 8437 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:44.198100 8437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:44.198256 8437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:44.199414 8437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "4870c23fc85a4efdbaf924cfa330fa92"
format_stamp: "Formatted at 2026-06-20 21:58:44 on dist-test-slave-gl2g"
server_key: "c75ba6f991dd4dacfdaf687b62cdc22b"
server_key_iv: "d74530080fbf4c62a64e3d27d7d46318"
server_key_version: "encryptionkey@0"
I20260620 21:58:44.199800 8437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "4870c23fc85a4efdbaf924cfa330fa92"
format_stamp: "Formatted at 2026-06-20 21:58:44 on dist-test-slave-gl2g"
server_key: "c75ba6f991dd4dacfdaf687b62cdc22b"
server_key_iv: "d74530080fbf4c62a64e3d27d7d46318"
server_key_version: "encryptionkey@0"
I20260620 21:58:44.201189 8437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 21:58:44.202008 8452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:44.202190 8437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:58:44.202299 8437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "4870c23fc85a4efdbaf924cfa330fa92"
format_stamp: "Formatted at 2026-06-20 21:58:44 on dist-test-slave-gl2g"
server_key: "c75ba6f991dd4dacfdaf687b62cdc22b"
server_key_iv: "d74530080fbf4c62a64e3d27d7d46318"
server_key_version: "encryptionkey@0"
I20260620 21:58:44.202376 8437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:44.217288 8437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:44.219466 8437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:44.219664 8437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:44.220297 8437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:44.220350 8437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:44.220391 8437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:44.220407 8437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:58:44 dist-test-slave-gl2g krb5kdc[8180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:44.227795 8437 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:46657
I20260620 21:58:44.227875 8567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:46657 every 8 connection(s)
I20260620 21:58:44.228273 8437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:58:44.230489 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.220743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:35533 (local address 127.6.241.190:37363)
0620 21:58:44.220813 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.220815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.221042 (+ 227us) server_negotiation.cc:407] Connection header received
0620 21:58:44.221536 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.221537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.221553 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.221643 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:44.222394 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.223285 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.224331 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.224679 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.224766 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.227603 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:44.227604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:44.227605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:44.227613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:44.229242 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.229690 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.229691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.229691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.229743 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.229970 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.229971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.229971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.230090 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:44.230143 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.230364 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.230426 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:58:44.230711 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.220737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:37363 (local address 127.6.241.130:35533)
0620 21:58:44.220885 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.220941 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.221438 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.221738 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.221743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.222065 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:44.222314 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.222320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.223385 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.223386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.224234 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.224236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.224791 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.224792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.224937 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.225489 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:44.225499 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:44.227460 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:44.229321 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.229325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.229327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.229620 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.229803 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.229804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.229804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.229910 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.230150 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:44.230153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:44.230272 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.230354 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.230479 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260620 21:58:44.231227 8568 heartbeater.cc:347] Connected to a master server at 127.6.241.190:37363
I20260620 21:58:44.231304 8568 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:44.231492 8568 heartbeater.cc:511] Master 127.6.241.190:37363 requested a full tablet report, sending...
I20260620 21:58:44.231869 8225 ts_manager.cc:194] Registered new tserver with Master: 4870c23fc85a4efdbaf924cfa330fa92 (127.6.241.130:46657)
I20260620 21:58:44.232350 8225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:35533
I20260620 21:58:44.237743 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.232261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:37363 (local address 127.6.241.130:48203)
0620 21:58:44.232411 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.232423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.232498 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.232625 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.232626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.232816 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:44.232873 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.232875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.233575 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.233575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.234097 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.234098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.234415 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.234415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.234470 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.234952 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:44.234957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:44.235267 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:44.236611 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.236612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.236613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.236775 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.236950 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.236951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.236951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.237002 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.237425 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:44.237426 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:44.237482 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.237554 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.237663 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260620 21:58:44.237805 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 8437
I20260620 21:58:44.237839 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.232314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:48203 (local address 127.6.241.190:37363)
0620 21:58:44.232352 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.232354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.232475 (+ 121us) server_negotiation.cc:407] Connection header received
0620 21:58:44.232549 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.232550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.232560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.232623 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:44.232946 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.233488 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.234169 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.234349 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.234402 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.235359 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:44.235360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:44.235360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:44.235366 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:44.236545 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.236837 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.236838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.236838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.236886 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.237157 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.237159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.237159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.237325 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:44.237393 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.237652 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.237739 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 21:58:44.237888 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:58:44.238055 7110 external_mini_cluster.cc:1428] Setting key ed718cd3bbf76786d785425148e7e801
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:58:44.263151 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:37363
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:44.347110 8577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:44.347383 8577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:44.347424 8577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:44.348747 8577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:44.348811 8577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:44.348845 8577 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:58:44.348897 8577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:58:44.350466 8577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.241.190:37363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:44.350775 8577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:44.351017 8577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:44.356207 8585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:44.356220 8582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:44.356220 8583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:44.356253 8577 server_base.cc:1034] running on GCE node
I20260620 21:58:44.356632 8577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:44.356870 8577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:44.358029 8577 hybrid_clock.cc:648] HybridClock initialized: now 1781992724358026 us; error 39 us; skew 500 ppm
Jun 20 21:58:44 dist-test-slave-gl2g krb5kdc[8180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:44.360570 8577 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:58:44.361073 8577 webserver.cc:460] Webserver started at http://127.6.241.131:44539/ using document root <none> and password file <none>
I20260620 21:58:44.361307 8577 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:44.361375 8577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:44.361539 8577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:44.362808 8577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb"
format_stamp: "Formatted at 2026-06-20 21:58:44 on dist-test-slave-gl2g"
server_key: "0d584cd2af03ebd846139207de824670"
server_key_iv: "f5e372a460f421f695912f58c69b5a97"
server_key_version: "encryptionkey@0"
I20260620 21:58:44.363184 8577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb"
format_stamp: "Formatted at 2026-06-20 21:58:44 on dist-test-slave-gl2g"
server_key: "0d584cd2af03ebd846139207de824670"
server_key_iv: "f5e372a460f421f695912f58c69b5a97"
server_key_version: "encryptionkey@0"
I20260620 21:58:44.364621 8577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:58:44.365372 8592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:44.365549 8577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:44.365646 8577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb"
format_stamp: "Formatted at 2026-06-20 21:58:44 on dist-test-slave-gl2g"
server_key: "0d584cd2af03ebd846139207de824670"
server_key_iv: "f5e372a460f421f695912f58c69b5a97"
server_key_version: "encryptionkey@0"
I20260620 21:58:44.365729 8577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:44.379323 8577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:44.381321 8577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:44.381500 8577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:44.382099 8577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:44.382159 8577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:44.382201 8577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:44.382223 8577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:58:44 dist-test-slave-gl2g krb5kdc[8180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:44.390877 8577 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:40459
I20260620 21:58:44.390966 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:40459 every 8 connection(s)
I20260620 21:58:44.391304 8577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:58:44.394479 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.382531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:40703 (local address 127.6.241.190:37363)
0620 21:58:44.382641 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.382643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.382953 (+ 310us) server_negotiation.cc:407] Connection header received
0620 21:58:44.383764 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.383766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.383782 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.383851 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:44.384953 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.385594 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.388024 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.388258 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.388336 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.391482 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:44.391484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:44.391485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:44.391495 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:44.392927 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.393410 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.393411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.393412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.393469 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.393754 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.393755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.393755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.393885 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:44.393966 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.394283 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.394382 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:58:44.394716 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.382522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:37363 (local address 127.6.241.131:40703)
0620 21:58:44.382785 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.382843 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.383606 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.383947 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.383951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.384418 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:44.384789 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.384797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.385695 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.385696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.387819 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.387824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.388360 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.388361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.388573 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.389328 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:44.389341 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:44.391342 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:44.393034 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.393048 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.393050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.393310 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.393541 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.393542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.393542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.393679 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.393955 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:44.393962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:44.394130 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.394249 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.394436 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":156,"threads_started":1}
I20260620 21:58:44.395280 8708 heartbeater.cc:347] Connected to a master server at 127.6.241.190:37363
I20260620 21:58:44.395401 8708 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:44.395615 8708 heartbeater.cc:511] Master 127.6.241.190:37363 requested a full tablet report, sending...
I20260620 21:58:44.396037 8225 ts_manager.cc:194] Registered new tserver with Master: f6ee5ba09cdb4942a7c85fd43323eeeb (127.6.241.131:40459)
I20260620 21:58:44.396628 8225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:40703
I20260620 21:58:44.398252 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 8577
I20260620 21:58:44.398361 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:58:44.398561 7110 external_mini_cluster.cc:1428] Setting key 277266f88529c1f26c39b82df4a86c5a
I20260620 21:58:44.399906 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:58:44.402782 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.396433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:37363 (local address 127.6.241.131:34677)
0620 21:58:44.396586 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.396595 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.396679 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.396899 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.396900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.397094 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:44.397147 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.397149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.397831 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.397833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.398546 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.398548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.398983 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.398985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.399084 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.399563 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:44.399568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:44.400082 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:44.401634 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.401636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.401636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.401894 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.402127 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:44.402127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:44.402128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:44.402179 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:44.402454 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:44.402454 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:44.402494 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.402543 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.402693 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260620 21:58:44.402786 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.396487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:34677 (local address 127.6.241.190:37363)
0620 21:58:44.396528 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.396532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.396655 (+ 123us) server_negotiation.cc:407] Connection header received
0620 21:58:44.396820 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.396821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.396834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.396894 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:44.397229 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.397729 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.398652 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.398907 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.398968 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.400175 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:44.400176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:44.400176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:44.400183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:44.401544 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.401981 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.401982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.401982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.402053 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.402239 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.402239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.402240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.402391 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:44.402470 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.402570 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.402688 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 21:58:44 dist-test-slave-gl2g krb5kdc[8180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:58:44 dist-test-slave-gl2g krb5kdc[8180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:44.424196 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.417037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.6.241.190:37363)
0620 21:58:44.417122 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.417125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.417135 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:58:44.417170 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.417170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.417180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.417234 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:44.417778 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.418280 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.419033 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.419225 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.419279 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.421572 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:44.421574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:44.421574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:44.421582 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:44.423058 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.423391 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.423393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.423393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.423455 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:44.423720 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:44.423721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:44.423722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:44.423849 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:44.423930 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.424033 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.424095 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:58:44.426085 8225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:44.427855 8225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:44.435051 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.430498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.6.241.130:46657)
0620 21:58:44.430670 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.430676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.430807 (+ 131us) server_negotiation.cc:407] Connection header received
0620 21:58:44.430929 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.430931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.430946 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.431048 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:44.431477 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.432633 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.434008 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.434519 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.434605 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.434707 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.434896 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.434940 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260620 21:58:44.435042 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.430515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:39511 (local address 127.0.0.1:56182)
0620 21:58:44.430920 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.430927 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.430978 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.431248 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.431251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.431253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:44.431334 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.431337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.432765 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.432766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.433751 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.433754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.434782 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.434783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.434832 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.434835 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.434879 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.434956 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":31,"thread_start_us":71,"threads_started":1}
I20260620 21:58:44.435073 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.430684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.6.241.129:39511)
0620 21:58:44.430835 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.430839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.430988 (+ 149us) server_negotiation.cc:407] Connection header received
0620 21:58:44.431135 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.431138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.431154 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.431238 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:44.431396 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.432682 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.433845 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.434549 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.434701 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.434755 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.434921 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.434959 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260620 21:58:44.435235 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.430405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.6.241.131:40459)
0620 21:58:44.431181 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.431186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.431195 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:58:44.431234 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.431236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.431253 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.431365 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:44.431540 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.432316 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.433754 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.434558 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.434644 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.434707 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.435088 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.435132 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":106,"threads_started":1}
I20260620 21:58:44.435732 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.430316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:46657 (local address 127.0.0.1:56572)
0620 21:58:44.430715 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.430743 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.430831 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.431347 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.431348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.431350 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:44.431411 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.431413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.432723 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.432724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.433937 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.433939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.434666 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.434667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.434773 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.434789 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.434879 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.435646 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":737,"spinlock_wait_cycles":768,"thread_start_us":82,"threads_started":1}
I20260620 21:58:44.435051 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.430316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:40459 (local address 127.0.0.1:60630)
0620 21:58:44.430697 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.430743 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.430813 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.431426 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.431427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.431429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:44.431477 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.431479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.432417 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.432418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.433663 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.433665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.434743 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.434744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.434809 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.434812 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.434864 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.434948 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":61,"threads_started":1}
I20260620 21:58:44.436427 8642 tablet_service.cc:1467] Processing CreateTablet for tablet f9f95e0ccf6f4a39a6a081b6c2582021 (DEFAULT_TABLE table=test-table [id=f17755ea004e42f8ab9f2be8d2607879]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:44.436434 8362 tablet_service.cc:1467] Processing CreateTablet for tablet f9f95e0ccf6f4a39a6a081b6c2582021 (DEFAULT_TABLE table=test-table [id=f17755ea004e42f8ab9f2be8d2607879]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:44.436770 8362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f95e0ccf6f4a39a6a081b6c2582021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:44.436772 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f95e0ccf6f4a39a6a081b6c2582021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:44.439525 8731 tablet_bootstrap.cc:492] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: Bootstrap starting.
I20260620 21:58:44.439894 8732 tablet_bootstrap.cc:492] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: Bootstrap starting.
I20260620 21:58:44.440574 8731 tablet_bootstrap.cc:654] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:44.440604 8732 tablet_bootstrap.cc:654] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:44.440616 8502 tablet_service.cc:1467] Processing CreateTablet for tablet f9f95e0ccf6f4a39a6a081b6c2582021 (DEFAULT_TABLE table=test-table [id=f17755ea004e42f8ab9f2be8d2607879]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:44.440936 8732 log.cc:824] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:44.440960 8731 log.cc:824] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:44.440999 8502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f95e0ccf6f4a39a6a081b6c2582021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:44.441767 8731 tablet_bootstrap.cc:492] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: No bootstrap required, opened a new log
I20260620 21:58:44.441843 8731 ts_tablet_manager.cc:1397] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:58:44.443078 8732 tablet_bootstrap.cc:492] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: No bootstrap required, opened a new log
I20260620 21:58:44.443171 8732 ts_tablet_manager.cc:1397] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 21:58:44.444007 8731 raft_consensus.cc:348] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } }
I20260620 21:58:44.444198 8731 raft_consensus.cc:374] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:44.444234 8731 raft_consensus.cc:729] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ee5ba09cdb4942a7c85fd43323eeeb, State: Initialized, Role: FOLLOWER
I20260620 21:58:44.444361 8731 consensus_queue.cc:260] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } }
I20260620 21:58:44.444655 8731 ts_tablet_manager.cc:1428] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 21:58:44.444774 8708 heartbeater.cc:503] Master 127.6.241.190:37363 was elected leader, sending a full tablet report...
I20260620 21:58:44.444842 8732 raft_consensus.cc:348] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } }
I20260620 21:58:44.445004 8732 raft_consensus.cc:374] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:44.445048 8732 raft_consensus.cc:729] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ba9f732db143e1a0bee8c2fe477086, State: Initialized, Role: FOLLOWER
I20260620 21:58:44.445168 8732 consensus_queue.cc:260] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } }
I20260620 21:58:44.445436 8732 ts_tablet_manager.cc:1428] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 21:58:44.445552 8428 heartbeater.cc:503] Master 127.6.241.190:37363 was elected leader, sending a full tablet report...
I20260620 21:58:44.445771 8736 tablet_bootstrap.cc:492] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: Bootstrap starting.
I20260620 21:58:44.446568 8736 tablet_bootstrap.cc:654] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:44.446914 8736 log.cc:824] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:44.447626 8736 tablet_bootstrap.cc:492] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: No bootstrap required, opened a new log
I20260620 21:58:44.447741 8736 ts_tablet_manager.cc:1397] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:58:44.449239 8736 raft_consensus.cc:348] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } }
I20260620 21:58:44.449406 8736 raft_consensus.cc:374] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:44.449460 8736 raft_consensus.cc:729] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4870c23fc85a4efdbaf924cfa330fa92, State: Initialized, Role: FOLLOWER
I20260620 21:58:44.449607 8736 consensus_queue.cc:260] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } }
I20260620 21:58:44.449887 8736 ts_tablet_manager.cc:1428] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 21:58:44.449985 8568 heartbeater.cc:503] Master 127.6.241.190:37363 was elected leader, sending a full tablet report...
W20260620 21:58:44.479121 8569 tablet.cc:2369] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:58:44.572964 8429 tablet.cc:2369] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:58:44.642019 8709 tablet.cc:2369] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:58:44.723996 8735 raft_consensus.cc:482] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:44.724138 8735 raft_consensus.cc:504] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } }
I20260620 21:58:44.724573 8735 leader_election.cc:288] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4870c23fc85a4efdbaf924cfa330fa92 (127.6.241.130:46657), 28ba9f732db143e1a0bee8c2fe477086 (127.6.241.129:39511)
I20260620 21:58:44.727695 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.724871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:56583 (local address 127.6.241.130:46657)
0620 21:58:44.724940 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.724942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.725143 (+ 201us) server_negotiation.cc:407] Connection header received
0620 21:58:44.725192 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.725194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.725209 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.725283 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:44.725455 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.725906 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.726942 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.727320 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.727412 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.727422 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.727574 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.727618 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:58:44.727715 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.724804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:46657 (local address 127.6.241.131:56583)
0620 21:58:44.725041 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.725057 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.725132 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.725286 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.725287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.725289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:44.725363 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.725365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.725990 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.725991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.726847 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.726849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.727411 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.727412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.727485 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.727489 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.727567 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.727620 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":99,"thread_start_us":95,"threads_started":1}
I20260620 21:58:44.727803 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.724921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:56441 (local address 127.6.241.129:39511)
0620 21:58:44.724977 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.724978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.724993 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:58:44.725037 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.725037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.725045 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.725130 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:44.725301 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.725792 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.726793 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.727374 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.727503 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.727512 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.727693 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.727740 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:58:44.727802 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.724804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:39511 (local address 127.6.241.131:56441)
0620 21:58:44.724875 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.724889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.724938 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.725136 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.725138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.725139 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:44.725215 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.725217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.725874 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.725875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.726702 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.726704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.727500 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.727501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.727599 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.727602 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.727685 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.727730 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 21:58:44.728152 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f95e0ccf6f4a39a6a081b6c2582021" candidate_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ba9f732db143e1a0bee8c2fe477086" is_pre_election: true
I20260620 21:58:44.728145 8522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f95e0ccf6f4a39a6a081b6c2582021" candidate_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4870c23fc85a4efdbaf924cfa330fa92" is_pre_election: true
I20260620 21:58:44.728327 8382 raft_consensus.cc:2454] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ee5ba09cdb4942a7c85fd43323eeeb in term 0.
I20260620 21:58:44.728327 8522 raft_consensus.cc:2454] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ee5ba09cdb4942a7c85fd43323eeeb in term 0.
I20260620 21:58:44.728601 8595 leader_election.cc:302] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4870c23fc85a4efdbaf924cfa330fa92, f6ee5ba09cdb4942a7c85fd43323eeeb; no voters:
I20260620 21:58:44.728772 8735 raft_consensus.cc:2781] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:44.728832 8735 raft_consensus.cc:482] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:44.728863 8735 raft_consensus.cc:3037] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:44.729995 8735 raft_consensus.cc:504] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } }
I20260620 21:58:44.730217 8735 leader_election.cc:288] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [CANDIDATE]: Term 1 election: Requested vote from peers 4870c23fc85a4efdbaf924cfa330fa92 (127.6.241.130:46657), 28ba9f732db143e1a0bee8c2fe477086 (127.6.241.129:39511)
I20260620 21:58:44.730329 8522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f95e0ccf6f4a39a6a081b6c2582021" candidate_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4870c23fc85a4efdbaf924cfa330fa92"
I20260620 21:58:44.730405 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f95e0ccf6f4a39a6a081b6c2582021" candidate_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ba9f732db143e1a0bee8c2fe477086"
I20260620 21:58:44.730440 8522 raft_consensus.cc:3037] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:44.730521 8382 raft_consensus.cc:3037] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:44.731400 8382 raft_consensus.cc:2454] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ee5ba09cdb4942a7c85fd43323eeeb in term 1.
I20260620 21:58:44.731467 8522 raft_consensus.cc:2454] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ee5ba09cdb4942a7c85fd43323eeeb in term 1.
I20260620 21:58:44.731655 8595 leader_election.cc:302] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4870c23fc85a4efdbaf924cfa330fa92, f6ee5ba09cdb4942a7c85fd43323eeeb; no voters:
I20260620 21:58:44.731750 8735 raft_consensus.cc:2781] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:44.731884 8735 raft_consensus.cc:686] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 LEADER]: Becoming Leader. State: Replica: f6ee5ba09cdb4942a7c85fd43323eeeb, State: Running, Role: LEADER
I20260620 21:58:44.732091 8735 consensus_queue.cc:237] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } }
I20260620 21:58:44.732995 8224 catalog_manager.cc:5495] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6ee5ba09cdb4942a7c85fd43323eeeb (127.6.241.131). New cstate: current_term: 1 leader_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } health_report { overall_health: UNKNOWN } } }
I20260620 21:58:44.789762 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.787017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.6.241.131:40459)
0620 21:58:44.787092 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.787094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.787103 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:58:44.787202 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.787203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.787219 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.787280 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:44.787438 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.787927 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.788847 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.789064 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.789131 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.789341 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:44.789437 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:44.789498 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.789557 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.789625 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:58:44.791679 8522 raft_consensus.cc:1261] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Refusing update from remote peer f6ee5ba09cdb4942a7c85fd43323eeeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:44.791703 8382 raft_consensus.cc:1261] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 FOLLOWER]: Refusing update from remote peer f6ee5ba09cdb4942a7c85fd43323eeeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:44.792039 8735 consensus_queue.cc:1035] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:44.792176 8741 consensus_queue.cc:1035] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:44.794811 8746 mvcc.cc:205] Tried to move back new op lower bound from 7299042200744017920 to 7299042200503373824. Current Snapshot: MvccSnapshot[applied={T|T < 7299042200744017920 or (T in {7299042200744017920})}]
I20260620 21:58:44.795145 8745 mvcc.cc:205] Tried to move back new op lower bound from 7299042200744017920 to 7299042200503373824. Current Snapshot: MvccSnapshot[applied={T|T < 7299042200744017920}]
I20260620 21:58:44.798733 8225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:44.798825 8225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:44.799711 8225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e: Sending DeleteTablet for 3 replicas of tablet f9f95e0ccf6f4a39a6a081b6c2582021
I20260620 21:58:44.800062 8502 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f95e0ccf6f4a39a6a081b6c2582021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:44 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:56572
I20260620 21:58:44.800060 8642 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f95e0ccf6f4a39a6a081b6c2582021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:44 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:60630
I20260620 21:58:44.800220 8362 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f95e0ccf6f4a39a6a081b6c2582021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:44 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:56182
I20260620 21:58:44.800294 8754 tablet_replica.cc:330] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: stopping tablet replica
I20260620 21:58:44.800398 8754 raft_consensus.cc:2229] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:44.800520 8754 raft_consensus.cc:2258] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:44.800490 8225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:44.800623 8225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:44.800839 8755 tablet_replica.cc:330] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: stopping tablet replica
I20260620 21:58:44.800945 8755 raft_consensus.cc:2229] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:44.801014 8754 ts_tablet_manager.cc:1905] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:44.801035 8755 raft_consensus.cc:2258] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:44.801448 8755 ts_tablet_manager.cc:1905] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:44.801945 8502 tablet_service.cc:1467] Processing CreateTablet for tablet 91d8ebf906cb4bb7a09612a5a5d55416 (DEFAULT_TABLE table=test-table [id=6af8bb9dbc3140fcb91b50a092af314e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:44.802059 8502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d8ebf906cb4bb7a09612a5a5d55416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:44.802769 8753 tablet_replica.cc:330] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: stopping tablet replica
I20260620 21:58:44.802842 8362 tablet_service.cc:1467] Processing CreateTablet for tablet 91d8ebf906cb4bb7a09612a5a5d55416 (DEFAULT_TABLE table=test-table [id=6af8bb9dbc3140fcb91b50a092af314e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:44.802871 8753 raft_consensus.cc:2229] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:44.802951 8362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d8ebf906cb4bb7a09612a5a5d55416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:44.802994 8753 raft_consensus.cc:2258] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:44.803236 8754 ts_tablet_manager.cc:1918] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:44.803299 8754 log.cc:1196] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/f9f95e0ccf6f4a39a6a081b6c2582021
I20260620 21:58:44.803553 8754 ts_tablet_manager.cc:1939] T f9f95e0ccf6f4a39a6a081b6c2582021 P 4870c23fc85a4efdbaf924cfa330fa92: Deleting consensus metadata
I20260620 21:58:44.804035 8755 ts_tablet_manager.cc:1918] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:44.804105 8755 log.cc:1196] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/f9f95e0ccf6f4a39a6a081b6c2582021
I20260620 21:58:44.804165 8211 catalog_manager.cc:4841] TS 4870c23fc85a4efdbaf924cfa330fa92 (127.6.241.130:46657): tablet f9f95e0ccf6f4a39a6a081b6c2582021 (table test-table [id=f17755ea004e42f8ab9f2be8d2607879]) successfully deleted
I20260620 21:58:44.804279 8732 tablet_bootstrap.cc:492] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: Bootstrap starting.
I20260620 21:58:44.804368 8755 ts_tablet_manager.cc:1939] T f9f95e0ccf6f4a39a6a081b6c2582021 P 28ba9f732db143e1a0bee8c2fe477086: Deleting consensus metadata
I20260620 21:58:44.804665 8732 tablet_bootstrap.cc:654] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:44.804764 8210 catalog_manager.cc:4841] TS 28ba9f732db143e1a0bee8c2fe477086 (127.6.241.129:39511): tablet f9f95e0ccf6f4a39a6a081b6c2582021 (table test-table [id=f17755ea004e42f8ab9f2be8d2607879]) successfully deleted
I20260620 21:58:44.804901 8642 tablet_service.cc:1467] Processing CreateTablet for tablet 91d8ebf906cb4bb7a09612a5a5d55416 (DEFAULT_TABLE table=test-table [id=6af8bb9dbc3140fcb91b50a092af314e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:44.805020 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d8ebf906cb4bb7a09612a5a5d55416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:44.805296 8732 tablet_bootstrap.cc:492] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: No bootstrap required, opened a new log
I20260620 21:58:44.805290 8753 ts_tablet_manager.cc:1905] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:44.805335 8732 ts_tablet_manager.cc:1397] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:44.805683 8732 raft_consensus.cc:348] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:44.805765 8732 raft_consensus.cc:374] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:44.805785 8732 raft_consensus.cc:729] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ba9f732db143e1a0bee8c2fe477086, State: Initialized, Role: FOLLOWER
I20260620 21:58:44.805823 8732 consensus_queue.cc:260] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:44.805913 8732 ts_tablet_manager.cc:1428] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:44.806067 8736 tablet_bootstrap.cc:492] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: Bootstrap starting.
I20260620 21:58:44.806432 8731 tablet_bootstrap.cc:492] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: Bootstrap starting.
I20260620 21:58:44.806460 8736 tablet_bootstrap.cc:654] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:44.806854 8731 tablet_bootstrap.cc:654] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:44.806972 8753 ts_tablet_manager.cc:1918] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:44.807039 8753 log.cc:1196] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/f9f95e0ccf6f4a39a6a081b6c2582021
I20260620 21:58:44.807288 8753 ts_tablet_manager.cc:1939] T f9f95e0ccf6f4a39a6a081b6c2582021 P f6ee5ba09cdb4942a7c85fd43323eeeb: Deleting consensus metadata
I20260620 21:58:44.807621 8736 tablet_bootstrap.cc:492] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: No bootstrap required, opened a new log
I20260620 21:58:44.807668 8736 ts_tablet_manager.cc:1397] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:44.807663 8210 catalog_manager.cc:4841] TS f6ee5ba09cdb4942a7c85fd43323eeeb (127.6.241.131:40459): tablet f9f95e0ccf6f4a39a6a081b6c2582021 (table test-table [id=f17755ea004e42f8ab9f2be8d2607879]) successfully deleted
I20260620 21:58:44.807929 8736 raft_consensus.cc:348] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:44.807986 8736 raft_consensus.cc:374] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:44.807999 8736 raft_consensus.cc:729] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4870c23fc85a4efdbaf924cfa330fa92, State: Initialized, Role: FOLLOWER
I20260620 21:58:44.808014 8731 tablet_bootstrap.cc:492] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: No bootstrap required, opened a new log
I20260620 21:58:44.808058 8731 ts_tablet_manager.cc:1397] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:58:44.808063 8736 consensus_queue.cc:260] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:44.808209 8736 ts_tablet_manager.cc:1428] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:58:44.808184 8731 raft_consensus.cc:348] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:44.808259 8731 raft_consensus.cc:374] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:44.808279 8731 raft_consensus.cc:729] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ee5ba09cdb4942a7c85fd43323eeeb, State: Initialized, Role: FOLLOWER
I20260620 21:58:44.808319 8731 consensus_queue.cc:260] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:44.808401 8731 ts_tablet_manager.cc:1428] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:44.964550 8737 raft_consensus.cc:482] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:44.964658 8737 raft_consensus.cc:504] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:44.965027 8737 leader_election.cc:288] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4870c23fc85a4efdbaf924cfa330fa92 (127.6.241.130:46657), f6ee5ba09cdb4942a7c85fd43323eeeb (127.6.241.131:40459)
I20260620 21:58:44.968398 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.965326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:40459 (local address 127.6.241.129:33423)
0620 21:58:44.965630 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.965641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.965711 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.965874 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.965875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.965876 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:44.965971 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.965972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.966578 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.966579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.967504 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.967507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.968118 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.968119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.968187 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.968191 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.968244 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.968296 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260620 21:58:44.968393 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.965405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:33423 (local address 127.6.241.131:40459)
0620 21:58:44.965463 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.965465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.965757 (+ 292us) server_negotiation.cc:407] Connection header received
0620 21:58:44.965785 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.965786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.965800 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.965862 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:44.966040 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.966480 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.967598 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.968013 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.968075 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.968083 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.968267 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.968318 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 21:58:44.968442 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.965517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:37371 (local address 127.6.241.130:46657)
0620 21:58:44.965606 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:44.965609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:44.965619 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:58:44.965653 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:44.965654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:44.965664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:44.965725 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:44.965930 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.966756 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.967677 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.968033 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.968130 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.968139 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:44.968321 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 21:58:44.968365 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:58:44.968541 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:44.965324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:46657 (local address 127.6.241.129:37371)
0620 21:58:44.965452 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:44.965469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:44.965525 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:44.965745 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:44.965746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:44.965748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:44.965845 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.965848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.966839 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.966839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.967588 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:44.967591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:44.968139 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:44.968140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:44.968219 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:44.968244 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:58:44.968302 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 21:58:44.968383 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260620 21:58:44.968693 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d8ebf906cb4bb7a09612a5a5d55416" candidate_uuid: "28ba9f732db143e1a0bee8c2fe477086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" is_pre_election: true
I20260620 21:58:44.968863 8662 raft_consensus.cc:2454] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ba9f732db143e1a0bee8c2fe477086 in term 0.
I20260620 21:58:44.968847 8522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d8ebf906cb4bb7a09612a5a5d55416" candidate_uuid: "28ba9f732db143e1a0bee8c2fe477086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4870c23fc85a4efdbaf924cfa330fa92" is_pre_election: true
I20260620 21:58:44.968928 8522 raft_consensus.cc:2454] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ba9f732db143e1a0bee8c2fe477086 in term 0.
I20260620 21:58:44.969071 8314 leader_election.cc:302] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ba9f732db143e1a0bee8c2fe477086, f6ee5ba09cdb4942a7c85fd43323eeeb; no voters:
I20260620 21:58:44.969208 8737 raft_consensus.cc:2781] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:44.969245 8737 raft_consensus.cc:482] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:44.969272 8737 raft_consensus.cc:3037] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:44.970016 8737 raft_consensus.cc:504] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:44.970160 8737 leader_election.cc:288] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [CANDIDATE]: Term 1 election: Requested vote from peers 4870c23fc85a4efdbaf924cfa330fa92 (127.6.241.130:46657), f6ee5ba09cdb4942a7c85fd43323eeeb (127.6.241.131:40459)
I20260620 21:58:44.970301 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d8ebf906cb4bb7a09612a5a5d55416" candidate_uuid: "28ba9f732db143e1a0bee8c2fe477086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb"
I20260620 21:58:44.970293 8522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d8ebf906cb4bb7a09612a5a5d55416" candidate_uuid: "28ba9f732db143e1a0bee8c2fe477086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4870c23fc85a4efdbaf924cfa330fa92"
I20260620 21:58:44.970368 8662 raft_consensus.cc:3037] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:44.970368 8522 raft_consensus.cc:3037] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:44.971153 8522 raft_consensus.cc:2454] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28ba9f732db143e1a0bee8c2fe477086 in term 1.
I20260620 21:58:44.971280 8662 raft_consensus.cc:2454] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28ba9f732db143e1a0bee8c2fe477086 in term 1.
I20260620 21:58:44.971465 8315 leader_election.cc:302] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ba9f732db143e1a0bee8c2fe477086, 4870c23fc85a4efdbaf924cfa330fa92; no voters:
I20260620 21:58:44.971623 8737 raft_consensus.cc:2781] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:44.971827 8737 raft_consensus.cc:686] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 LEADER]: Becoming Leader. State: Replica: 28ba9f732db143e1a0bee8c2fe477086, State: Running, Role: LEADER
I20260620 21:58:44.971912 8737 consensus_queue.cc:237] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:44.972635 8224 catalog_manager.cc:5495] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28ba9f732db143e1a0bee8c2fe477086 (127.6.241.129). New cstate: current_term: 1 leader_uuid: "28ba9f732db143e1a0bee8c2fe477086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } health_report { overall_health: UNKNOWN } } }
Jun 20 21:58:45 dist-test-slave-gl2g krb5kdc[8180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992723, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:45.023447 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:45.015396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:37363 (local address 127.0.0.1:47030)
0620 21:58:45.015663 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:45.015674 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:45.015776 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:45.015956 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:45.015957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:45.016312 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:45.016360 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:45.016363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.017059 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.017060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:45.017593 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:45.017595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.017990 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.017996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:45.018053 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:45.018573 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:45.018578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:45.020200 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:45.022311 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:45.022322 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:45.022325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:45.022653 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:45.022863 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:45.022864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:45.022864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:45.022924 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:45.023160 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:45.023167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:45.023211 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 21:58:45.023264 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:58:45.023367 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":190,"threads_started":1}
I20260620 21:58:45.023521 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:45.015419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.6.241.190:37363)
0620 21:58:45.015651 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:45.015654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:45.015757 (+ 103us) server_negotiation.cc:407] Connection header received
0620 21:58:45.015852 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:45.015852 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:45.015865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:45.015954 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:45.016462 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.016966 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.017694 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.017885 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.017975 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:45.020330 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:45.020332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:45.020332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:45.020339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:45.022218 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:45.022725 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:45.022726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:45.022726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:45.022793 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:45.022979 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:45.022980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:45.022980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:45.023099 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:45.023155 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:45.023313 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 21:58:45.023427 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":204,"thread_start_us":168,"threads_started":1}
I20260620 21:58:45.025666 8224 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:47030:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:58:45.025871 8224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:45.027482 8502 tablet_service.cc:1467] Processing CreateTablet for tablet 26f293deed6344aea3473587dc433b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0a725202b0c424a9f3c2cbe1ea92811]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:58:45.027518 8362 tablet_service.cc:1467] Processing CreateTablet for tablet 26f293deed6344aea3473587dc433b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0a725202b0c424a9f3c2cbe1ea92811]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:58:45.027539 8642 tablet_service.cc:1467] Processing CreateTablet for tablet 26f293deed6344aea3473587dc433b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0a725202b0c424a9f3c2cbe1ea92811]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:58:45.027670 8502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f293deed6344aea3473587dc433b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:45.027671 8362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f293deed6344aea3473587dc433b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:45.027679 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f293deed6344aea3473587dc433b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:45.028975 8736 tablet_bootstrap.cc:492] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: Bootstrap starting.
I20260620 21:58:45.029234 8732 tablet_bootstrap.cc:492] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: Bootstrap starting.
I20260620 21:58:45.029233 8731 tablet_bootstrap.cc:492] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: Bootstrap starting.
I20260620 21:58:45.029448 8736 tablet_bootstrap.cc:654] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:45.029562 8731 tablet_bootstrap.cc:654] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:45.029713 8732 tablet_bootstrap.cc:654] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:45.030193 8736 tablet_bootstrap.cc:492] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: No bootstrap required, opened a new log
I20260620 21:58:45.030251 8736 ts_tablet_manager.cc:1397] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:45.030334 8731 tablet_bootstrap.cc:492] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: No bootstrap required, opened a new log
I20260620 21:58:45.030381 8731 ts_tablet_manager.cc:1397] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:58:45.030404 8736 raft_consensus.cc:348] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:45.030483 8736 raft_consensus.cc:374] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:45.030504 8736 raft_consensus.cc:729] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4870c23fc85a4efdbaf924cfa330fa92, State: Initialized, Role: FOLLOWER
I20260620 21:58:45.030514 8731 raft_consensus.cc:348] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:45.030584 8731 raft_consensus.cc:374] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:45.030639 8731 raft_consensus.cc:729] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ee5ba09cdb4942a7c85fd43323eeeb, State: Initialized, Role: FOLLOWER
I20260620 21:58:45.030558 8736 consensus_queue.cc:260] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:45.030680 8731 consensus_queue.cc:260] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:45.030745 8736 ts_tablet_manager.cc:1428] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:45.030778 8731 ts_tablet_manager.cc:1428] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:45.030807 8741 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.030921 8741 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:45.030900 8739 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.030984 8739 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:45.031335 8732 tablet_bootstrap.cc:492] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: No bootstrap required, opened a new log
I20260620 21:58:45.031385 8732 ts_tablet_manager.cc:1397] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:45.031523 8732 raft_consensus.cc:348] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:45.031621 8732 raft_consensus.cc:374] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:45.031654 8732 raft_consensus.cc:729] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ba9f732db143e1a0bee8c2fe477086, State: Initialized, Role: FOLLOWER
I20260620 21:58:45.031695 8732 consensus_queue.cc:260] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:45.031788 8732 ts_tablet_manager.cc:1428] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:45.031850 8737 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.031958 8737 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:45.268674 8737 raft_consensus.cc:482] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:45.268826 8737 raft_consensus.cc:504] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:45.269061 8737 leader_election.cc:288] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4870c23fc85a4efdbaf924cfa330fa92 (127.6.241.130:46657), f6ee5ba09cdb4942a7c85fd43323eeeb (127.6.241.131:40459)
I20260620 21:58:45.269323 8522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f293deed6344aea3473587dc433b75" candidate_uuid: "28ba9f732db143e1a0bee8c2fe477086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4870c23fc85a4efdbaf924cfa330fa92" is_pre_election: true
I20260620 21:58:45.269323 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f293deed6344aea3473587dc433b75" candidate_uuid: "28ba9f732db143e1a0bee8c2fe477086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" is_pre_election: true
I20260620 21:58:45.269434 8522 raft_consensus.cc:2454] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ba9f732db143e1a0bee8c2fe477086 in term 0.
I20260620 21:58:45.269434 8662 raft_consensus.cc:2454] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ba9f732db143e1a0bee8c2fe477086 in term 0.
I20260620 21:58:45.269577 8315 leader_election.cc:302] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ba9f732db143e1a0bee8c2fe477086, 4870c23fc85a4efdbaf924cfa330fa92; no voters:
I20260620 21:58:45.269701 8737 raft_consensus.cc:2781] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:45.269755 8737 raft_consensus.cc:482] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:45.269784 8737 raft_consensus.cc:3037] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:45.270512 8737 raft_consensus.cc:504] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:45.270686 8737 leader_election.cc:288] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [CANDIDATE]: Term 1 election: Requested vote from peers 4870c23fc85a4efdbaf924cfa330fa92 (127.6.241.130:46657), f6ee5ba09cdb4942a7c85fd43323eeeb (127.6.241.131:40459)
I20260620 21:58:45.270851 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f293deed6344aea3473587dc433b75" candidate_uuid: "28ba9f732db143e1a0bee8c2fe477086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb"
I20260620 21:58:45.270861 8522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f293deed6344aea3473587dc433b75" candidate_uuid: "28ba9f732db143e1a0bee8c2fe477086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4870c23fc85a4efdbaf924cfa330fa92"
I20260620 21:58:45.270941 8522 raft_consensus.cc:3037] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:45.270941 8662 raft_consensus.cc:3037] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:45.271711 8662 raft_consensus.cc:2454] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28ba9f732db143e1a0bee8c2fe477086 in term 1.
I20260620 21:58:45.271711 8522 raft_consensus.cc:2454] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28ba9f732db143e1a0bee8c2fe477086 in term 1.
I20260620 21:58:45.271908 8315 leader_election.cc:302] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ba9f732db143e1a0bee8c2fe477086, 4870c23fc85a4efdbaf924cfa330fa92; no voters:
I20260620 21:58:45.272011 8737 raft_consensus.cc:2781] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:45.272071 8737 raft_consensus.cc:686] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 LEADER]: Becoming Leader. State: Replica: 28ba9f732db143e1a0bee8c2fe477086, State: Running, Role: LEADER
I20260620 21:58:45.272133 8737 consensus_queue.cc:237] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } }
I20260620 21:58:45.272395 8761 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: TxnStatusTablet state changed. Reason: New leader 28ba9f732db143e1a0bee8c2fe477086. Latest consensus state: current_term: 1 leader_uuid: "28ba9f732db143e1a0bee8c2fe477086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.272491 8761 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:58:45.272694 8774 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:58:45.272810 8774 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:58:45.272728 8224 catalog_manager.cc:5495] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28ba9f732db143e1a0bee8c2fe477086 (127.6.241.129). New cstate: current_term: 1 leader_uuid: "28ba9f732db143e1a0bee8c2fe477086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260620 21:58:45.318571 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:45.315860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.6.241.129:39511)
0620 21:58:45.315981 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:45.315983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:45.315992 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:58:45.316036 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:45.316037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:45.316057 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:45.316150 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:45.316302 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.316863 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.317636 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.317864 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.317956 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:45.318183 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:45.318302 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:45.318389 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:45.318446 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 21:58:45.318469 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":51,"threads_started":1}
I20260620 21:58:45.318567 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:45.315774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:39511 (local address 127.0.0.1:56192)
0620 21:58:45.315853 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:45.315867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:45.315923 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:45.316154 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:45.316155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:45.316156 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:58:45.316217 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:45.316219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.316952 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.316954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:45.317541 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:45.317543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.317958 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.317959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:45.318034 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:45.318047 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:58:45.318372 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:58:45.318383 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 21:58:45.318439 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 21:58:45.318496 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 21:58:45.319970 8522 raft_consensus.cc:1261] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Refusing update from remote peer 28ba9f732db143e1a0bee8c2fe477086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:45.320030 8662 raft_consensus.cc:1261] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 FOLLOWER]: Refusing update from remote peer 28ba9f732db143e1a0bee8c2fe477086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:45.320278 8761 consensus_queue.cc:1035] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:45.320406 8737 consensus_queue.cc:1035] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:45.321408 8739 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: TxnStatusTablet state changed. Reason: New leader 28ba9f732db143e1a0bee8c2fe477086. Latest consensus state: current_term: 1 leader_uuid: "28ba9f732db143e1a0bee8c2fe477086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.321516 8739 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:45.321575 8741 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: TxnStatusTablet state changed. Reason: New leader 28ba9f732db143e1a0bee8c2fe477086. Latest consensus state: current_term: 1 leader_uuid: "28ba9f732db143e1a0bee8c2fe477086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.321666 8741 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:45.322443 8737 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28ba9f732db143e1a0bee8c2fe477086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.322531 8737 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:58:45.322646 8737 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28ba9f732db143e1a0bee8c2fe477086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.322640 8739 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28ba9f732db143e1a0bee8c2fe477086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.322703 8737 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P 28ba9f732db143e1a0bee8c2fe477086: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:58:45.322722 8739 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P 4870c23fc85a4efdbaf924cfa330fa92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:45.323243 8741 tablet_replica.cc:439] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28ba9f732db143e1a0bee8c2fe477086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ba9f732db143e1a0bee8c2fe477086" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 39511 } } peers { permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 } } peers { permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 } } }
I20260620 21:58:45.323333 8741 tablet_replica.cc:442] T 26f293deed6344aea3473587dc433b75 P f6ee5ba09cdb4942a7c85fd43323eeeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:45.327148 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:45.324619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.6.241.129:39511)
0620 21:58:45.324662 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:45.324664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:45.324706 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:58:45.324777 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:45.324778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:45.324789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:45.324846 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:45.324934 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.325674 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.326402 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.326713 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.326780 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:45.326918 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:45.326955 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:45.327005 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:45.327055 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 21:58:45.327076 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:58:45.332218 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:45.329579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:44323 (local address 127.6.241.129:39511)
0620 21:58:45.329625 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:45.329627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:45.329733 (+ 106us) server_negotiation.cc:407] Connection header received
0620 21:58:45.329756 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:45.329757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:45.329767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:45.329832 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:45.329956 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.330698 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.331444 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.331728 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.331799 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:45.331979 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:45.332010 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:45.332062 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:45.332111 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 21:58:45.332130 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 21:58:45.332222 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:45.329514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:39511 (local address 127.6.241.129:44323)
0620 21:58:45.329651 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:45.329662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:45.329722 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:45.329834 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:45.329834 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:45.329835 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:58:45.329871 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:45.329872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.330765 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.330766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:45.331371 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:45.331373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.331807 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.331808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:45.331848 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:45.331860 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:58:45.332070 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:58:45.332072 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:58:45.332107 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 21:58:45.332142 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260620 21:58:45.334868 8522 raft_consensus.cc:1261] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Refusing update from remote peer 28ba9f732db143e1a0bee8c2fe477086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:45.334900 8662 raft_consensus.cc:1261] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 FOLLOWER]: Refusing update from remote peer 28ba9f732db143e1a0bee8c2fe477086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:45.335114 8737 consensus_queue.cc:1035] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4870c23fc85a4efdbaf924cfa330fa92" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:45.335246 8761 consensus_queue.cc:1035] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ee5ba09cdb4942a7c85fd43323eeeb" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:45.347268 8224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:45.347391 8224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:45.348099 8224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5715b077ccf4a97afc94cb8365dc75e: Sending DeleteTablet for 3 replicas of tablet 91d8ebf906cb4bb7a09612a5a5d55416
I20260620 21:58:45.348364 8642 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d8ebf906cb4bb7a09612a5a5d55416 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:45 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:60630
I20260620 21:58:45.348600 8795 tablet_replica.cc:330] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: stopping tablet replica
I20260620 21:58:45.348594 8502 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d8ebf906cb4bb7a09612a5a5d55416 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:45 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:56572
I20260620 21:58:45.348662 8795 raft_consensus.cc:2229] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:45.348711 8795 raft_consensus.cc:2258] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:45.348778 8361 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d8ebf906cb4bb7a09612a5a5d55416 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:45 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:56182
I20260620 21:58:45.348990 8795 ts_tablet_manager.cc:1905] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:45.348981 8797 tablet_replica.cc:330] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: stopping tablet replica
I20260620 21:58:45.349152 8797 raft_consensus.cc:2229] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:45.349299 8797 raft_consensus.cc:2258] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:45.349593 8797 ts_tablet_manager.cc:1905] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:45.349635 8796 tablet_replica.cc:330] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: stopping tablet replica
I20260620 21:58:45.349694 8796 raft_consensus.cc:2229] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:45.349736 8796 raft_consensus.cc:2258] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:45.349974 8796 ts_tablet_manager.cc:1905] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:45.351022 8795 ts_tablet_manager.cc:1918] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:58:45.351086 8795 log.cc:1196] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/91d8ebf906cb4bb7a09612a5a5d55416
I20260620 21:58:45.351318 8795 ts_tablet_manager.cc:1939] T 91d8ebf906cb4bb7a09612a5a5d55416 P f6ee5ba09cdb4942a7c85fd43323eeeb: Deleting consensus metadata
I20260620 21:58:45.351647 8210 catalog_manager.cc:4841] TS f6ee5ba09cdb4942a7c85fd43323eeeb (127.6.241.131:40459): tablet 91d8ebf906cb4bb7a09612a5a5d55416 (table test-table [id=6af8bb9dbc3140fcb91b50a092af314e]) successfully deleted
I20260620 21:58:45.351711 8796 ts_tablet_manager.cc:1918] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:58:45.351763 8796 log.cc:1196] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/91d8ebf906cb4bb7a09612a5a5d55416
I20260620 21:58:45.351939 8796 ts_tablet_manager.cc:1939] T 91d8ebf906cb4bb7a09612a5a5d55416 P 4870c23fc85a4efdbaf924cfa330fa92: Deleting consensus metadata
I20260620 21:58:45.351948 8797 ts_tablet_manager.cc:1918] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:58:45.352054 8797 log.cc:1196] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/91d8ebf906cb4bb7a09612a5a5d55416
I20260620 21:58:45.352285 8797 ts_tablet_manager.cc:1939] T 91d8ebf906cb4bb7a09612a5a5d55416 P 28ba9f732db143e1a0bee8c2fe477086: Deleting consensus metadata
I20260620 21:58:45.352383 8211 catalog_manager.cc:4841] TS 4870c23fc85a4efdbaf924cfa330fa92 (127.6.241.130:46657): tablet 91d8ebf906cb4bb7a09612a5a5d55416 (table test-table [id=6af8bb9dbc3140fcb91b50a092af314e]) successfully deleted
I20260620 21:58:45.352548 8210 catalog_manager.cc:4841] TS 28ba9f732db143e1a0bee8c2fe477086 (127.6.241.129:39511): tablet 91d8ebf906cb4bb7a09612a5a5d55416 (table test-table [id=6af8bb9dbc3140fcb91b50a092af314e]) successfully deleted
Jun 20 21:58:45 dist-test-slave-gl2g krb5kdc[8180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:58:45.361578 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:45.352798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.6.241.129:39511)
0620 21:58:45.352839 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:45.352841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:45.352988 (+ 147us) server_negotiation.cc:407] Connection header received
0620 21:58:45.353067 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:45.353068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:45.353079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:45.353134 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:45.353788 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.354271 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.355075 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.355280 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.355366 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:45.358254 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:45.358262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:45.358263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:45.358271 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:45.360330 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:45.360714 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:45.360718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:45.360718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:45.360802 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:45.361002 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:45.361003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:45.361003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:45.361193 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:45.361342 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:45.361395 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 21:58:45.361482 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 21:58:45.361974 8404 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56208
I20260620 21:58:45.369863 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:45.363232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.6.241.190:37363)
0620 21:58:45.363285 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:45.363287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:45.363332 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:58:45.363382 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:45.363383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:45.363397 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:45.363469 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:45.363990 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.364799 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.365492 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:45.365816 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:45.365890 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:45.367107 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:45.367108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:45.367108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:45.367116 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:45.368641 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:45.369029 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:45.369030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:45.369030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:45.369096 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:45.369336 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:45.369337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:45.369337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:45.369481 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:45.369591 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:45.369671 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 21:58:45.369767 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 21:58:45.370114 8224 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47038
I20260620 21:58:45.370832 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 8297
I20260620 21:58:45.375751 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 8437
I20260620 21:58:45.380527 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 8577
I20260620 21:58:45.385883 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 8193
2026-06-20T21:58:45Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2830 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:58:45 dist-test-slave-gl2g krb5kdc[8804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:58:45 dist-test-slave-gl2g krb5kdc[8804](info): set up 2 sockets
Jun 20 21:58:45 dist-test-slave-gl2g krb5kdc[8804](info): commencing operation
krb5kdc: starting...
W20260620 21:58:46.426420 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:58:46 dist-test-slave-gl2g krb5kdc[8804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:58:46.549945 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40053
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:42167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:46.632328 8817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:46.632611 8817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:46.632651 8817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:46.633877 8817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:46.633960 8817 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:58:46.633983 8817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:46.634006 8817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:46.634035 8817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:46.634061 8817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:46.635617 8817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:42167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:40053
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:46.635916 8817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:46.636155 8817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:46.641213 8823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:46.641288 8817 server_base.cc:1034] running on GCE node
W20260620 21:58:46.641327 8822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:46.641213 8825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:46.641748 8817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:46.642100 8817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:46.643265 8817 hybrid_clock.cc:648] HybridClock initialized: now 1781992726643235 us; error 47 us; skew 500 ppm
Jun 20 21:58:46 dist-test-slave-gl2g krb5kdc[8804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992726, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:46.645903 8817 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:58:46.646389 8817 webserver.cc:460] Webserver started at http://127.6.241.190:33875/ using document root <none> and password file <none>
I20260620 21:58:46.646672 8817 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:46.646724 8817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:46.646868 8817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:46.647987 8817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "1a36630912734c67a02e32c963f9774c"
format_stamp: "Formatted at 2026-06-20 21:58:46 on dist-test-slave-gl2g"
server_key: "4ae5d75e7a930e12c9616bd35101ba04"
server_key_iv: "659a9f25224ebf6f3bb223fceab2442d"
server_key_version: "encryptionkey@0"
I20260620 21:58:46.648375 8817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "1a36630912734c67a02e32c963f9774c"
format_stamp: "Formatted at 2026-06-20 21:58:46 on dist-test-slave-gl2g"
server_key: "4ae5d75e7a930e12c9616bd35101ba04"
server_key_iv: "659a9f25224ebf6f3bb223fceab2442d"
server_key_version: "encryptionkey@0"
I20260620 21:58:46.649811 8817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:58:46.650674 8832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:46.650833 8817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:46.650930 8817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "1a36630912734c67a02e32c963f9774c"
format_stamp: "Formatted at 2026-06-20 21:58:46 on dist-test-slave-gl2g"
server_key: "4ae5d75e7a930e12c9616bd35101ba04"
server_key_iv: "659a9f25224ebf6f3bb223fceab2442d"
server_key_version: "encryptionkey@0"
I20260620 21:58:46.650987 8817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:46.680169 8817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:46.682765 8817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:46.682888 8817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:46.687675 8817 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:40053
I20260620 21:58:46.687721 8894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:40053 every 8 connection(s)
I20260620 21:58:46.688086 8817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:46.688831 8895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:46.691323 8895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c: Bootstrap starting.
I20260620 21:58:46.691991 8895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:46.692320 8895 log.cc:824] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:46.693197 8895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c: No bootstrap required, opened a new log
I20260620 21:58:46.695438 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 8817
I20260620 21:58:46.695533 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:58:46.695722 7110 external_mini_cluster.cc:1428] Setting key 60cffd7450b92438e34b41f97b2b902e
I20260620 21:58:46.695636 8895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a36630912734c67a02e32c963f9774c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40053 } }
I20260620 21:58:46.695835 8895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:46.695886 8895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a36630912734c67a02e32c963f9774c, State: Initialized, Role: FOLLOWER
I20260620 21:58:46.696478 8895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a36630912734c67a02e32c963f9774c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40053 } }
I20260620 21:58:46.696568 8895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:46.696624 8895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:46.696681 8895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:46.697556 8895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a36630912734c67a02e32c963f9774c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40053 } }
I20260620 21:58:46.697698 8895 leader_election.cc:302] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a36630912734c67a02e32c963f9774c; no voters:
I20260620 21:58:46.697896 8895 leader_election.cc:288] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:58:46.697923 8900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:46.698058 8900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [term 1 LEADER]: Becoming Leader. State: Replica: 1a36630912734c67a02e32c963f9774c, State: Running, Role: LEADER
I20260620 21:58:46.698163 8900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a36630912734c67a02e32c963f9774c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40053 } }
I20260620 21:58:46.698200 8895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:46.699131 8901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a36630912734c67a02e32c963f9774c. Latest consensus state: current_term: 1 leader_uuid: "1a36630912734c67a02e32c963f9774c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a36630912734c67a02e32c963f9774c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40053 } } }
I20260620 21:58:46.699232 8901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:46.699694 8902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a36630912734c67a02e32c963f9774c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a36630912734c67a02e32c963f9774c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40053 } } }
I20260620 21:58:46.699781 8902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:46.700100 8911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:46.700558 8911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:46.702371 8911 catalog_manager.cc:1347] Generated new cluster ID: 33004bb9b39e40688190b731700d63bc
I20260620 21:58:46.702435 8911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 21:58:46 dist-test-slave-gl2g krb5kdc[8804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:46.708199 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:46.696266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.6.241.190:40053)
0620 21:58:46.696507 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:46.696516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:46.696544 (+ 28us) server_negotiation.cc:407] Connection header received
0620 21:58:46.696725 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:46.696732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:46.696797 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:46.696924 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:46.697788 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.699229 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.700351 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.700975 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.701089 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:46.704121 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:46.704126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:46.704126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:46.704140 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:46.706249 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:46.706821 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:46.706824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:46.706825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:46.706971 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:46.707201 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:46.707202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:46.707202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:46.707523 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:46.707689 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:46.707821 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 21:58:46.707979 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":114,"threads_started":1}
I20260620 21:58:46.718461 8911 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:58:46.719065 8911 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:46.724689 8911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c: Generated new TSK 0
I20260620 21:58:46.724915 8911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:58:46.753793 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40053
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:42167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:46.838467 8921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:46.838706 8921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:46.838739 8921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:46.839934 8921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:46.839974 8921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:46.839995 8921 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:58:46.840008 8921 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:58:46.840025 8921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:58:46.841475 8921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:42167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.241.190:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:46.841691 8921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:46.841877 8921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:46.847448 8927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:46.847455 8929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:46.847448 8926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:46.847709 8921 server_base.cc:1034] running on GCE node
I20260620 21:58:46.847895 8921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:46.848148 8921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:46.849314 8921 hybrid_clock.cc:648] HybridClock initialized: now 1781992726849296 us; error 46 us; skew 500 ppm
Jun 20 21:58:46 dist-test-slave-gl2g krb5kdc[8804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992726, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:46.851984 8921 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:58:46.852458 8921 webserver.cc:460] Webserver started at http://127.6.241.129:36891/ using document root <none> and password file <none>
I20260620 21:58:46.852720 8921 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:46.852792 8921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:46.852941 8921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:46.854084 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "b049c8424a7346189c4e551d2f4abadd"
format_stamp: "Formatted at 2026-06-20 21:58:46 on dist-test-slave-gl2g"
server_key: "986d69401718d7ee91e5a58f1c66a573"
server_key_iv: "0afdf1fb6ecfb7c4d1486bbdd913c7b3"
server_key_version: "encryptionkey@0"
I20260620 21:58:46.854458 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "b049c8424a7346189c4e551d2f4abadd"
format_stamp: "Formatted at 2026-06-20 21:58:46 on dist-test-slave-gl2g"
server_key: "986d69401718d7ee91e5a58f1c66a573"
server_key_iv: "0afdf1fb6ecfb7c4d1486bbdd913c7b3"
server_key_version: "encryptionkey@0"
I20260620 21:58:46.855995 8921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:58:46.856769 8936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:46.856951 8921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:46.857054 8921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "b049c8424a7346189c4e551d2f4abadd"
format_stamp: "Formatted at 2026-06-20 21:58:46 on dist-test-slave-gl2g"
server_key: "986d69401718d7ee91e5a58f1c66a573"
server_key_iv: "0afdf1fb6ecfb7c4d1486bbdd913c7b3"
server_key_version: "encryptionkey@0"
I20260620 21:58:46.857143 8921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:46.870285 8921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:46.872679 8921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:46.872874 8921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:46.873428 8921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:46.873490 8921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:46.873525 8921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:46.873555 8921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:58:46 dist-test-slave-gl2g krb5kdc[8804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992726, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:46.881235 8921 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:41151
I20260620 21:58:46.881383 9051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:41151 every 8 connection(s)
I20260620 21:58:46.881670 8921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:58:46.884835 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:46.874126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:34615 (local address 127.6.241.190:40053)
0620 21:58:46.874175 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:46.874177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:46.874456 (+ 279us) server_negotiation.cc:407] Connection header received
0620 21:58:46.875133 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:46.875135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:46.875148 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:46.875196 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:46.875968 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.876870 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.878706 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.879042 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.879140 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:46.881799 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:46.881801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:46.881802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:46.881811 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:46.883413 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:46.883868 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:46.883870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:46.883870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:46.883934 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:46.884179 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:46.884179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:46.884179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:46.884311 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:46.884368 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:46.884688 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 21:58:46.884766 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 21:58:46.884915 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:46.874133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40053 (local address 127.6.241.129:34615)
0620 21:58:46.874319 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:46.874359 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:46.875012 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:46.875272 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:46.875277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:46.875621 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:46.875860 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:46.875866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.876986 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.876988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:46.878572 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:46.878577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.879159 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.879160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:46.879332 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:46.880065 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:46.880083 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:46.881681 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:46.883505 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:46.883513 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:46.883514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:46.883768 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:46.884004 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:46.884004 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:46.884005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:46.884118 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:46.884369 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:46.884374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:46.884502 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 21:58:46.884594 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 21:58:46.884737 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260620 21:58:46.885496 9052 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40053
I20260620 21:58:46.885591 9052 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:46.885803 9052 heartbeater.cc:511] Master 127.6.241.190:40053 requested a full tablet report, sending...
I20260620 21:58:46.886437 8848 ts_manager.cc:194] Registered new tserver with Master: b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151)
I20260620 21:58:46.887413 8848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:34615
I20260620 21:58:46.889014 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 8921
I20260620 21:58:46.889142 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:58:46.889353 7110 external_mini_cluster.cc:1428] Setting key b247436a3d32fdc4bbcf8fa5364c8f59
I20260620 21:58:46.893246 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:46.886526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40053 (local address 127.6.241.129:41939)
0620 21:58:46.886690 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:46.886703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:46.886752 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:46.886908 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:46.886909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:46.887117 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:46.887183 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:46.887185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.887993 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.887994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:46.888693 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:46.888696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.889086 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.889088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:46.889182 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:46.889761 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:46.889766 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:46.890185 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:46.892128 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:46.892129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:46.892130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:46.892391 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:46.892596 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:46.892597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:46.892597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:46.892671 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:46.892905 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:46.892906 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:46.892967 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 21:58:46.893023 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 21:58:46.893172 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":19,"thread_start_us":93,"threads_started":1}
I20260620 21:58:46.893296 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:46.886727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:41939 (local address 127.6.241.190:40053)
0620 21:58:46.886785 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:46.886787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:46.886795 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:58:46.886816 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:46.886816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:46.886827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:46.886884 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:46.887257 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.887907 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.888801 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:46.888995 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:46.889064 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:46.890272 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:46.890274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:46.890274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:46.890282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:46.892030 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:46.892473 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:46.892474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:46.892474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:46.892528 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:46.892733 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:46.892733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:46.892734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:46.892841 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:46.892890 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:46.893072 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 21:58:46.893163 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:58:46.917963 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40053
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:42167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:47.002210 9061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:47.002456 9061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:47.002486 9061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:47.003851 9061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:47.003907 9061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:47.003931 9061 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:58:47.003944 9061 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:58:47.003963 9061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:58:47.005563 9061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:42167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.241.190:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:47.005848 9061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:47.006105 9061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:47.011636 9066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:47.011669 9069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:47.011624 9061 server_base.cc:1034] running on GCE node
W20260620 21:58:47.011626 9067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:47.012013 9061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:47.012292 9061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:47.013495 9061 hybrid_clock.cc:648] HybridClock initialized: now 1781992727013470 us; error 54 us; skew 500 ppm
Jun 20 21:58:47 dist-test-slave-gl2g krb5kdc[8804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992727, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:47.016302 9061 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:58:47.016820 9061 webserver.cc:460] Webserver started at http://127.6.241.130:33861/ using document root <none> and password file <none>
I20260620 21:58:47.017068 9061 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:47.017143 9061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:47.017294 9061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:47.018390 9061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "b4f5eb72922b40c191865b03dc9ce6c7"
format_stamp: "Formatted at 2026-06-20 21:58:47 on dist-test-slave-gl2g"
server_key: "5838199c7ee1efffed7eaecb7f81facc"
server_key_iv: "90d329b0f3d078aace9b29ffc3094653"
server_key_version: "encryptionkey@0"
I20260620 21:58:47.018796 9061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "b4f5eb72922b40c191865b03dc9ce6c7"
format_stamp: "Formatted at 2026-06-20 21:58:47 on dist-test-slave-gl2g"
server_key: "5838199c7ee1efffed7eaecb7f81facc"
server_key_iv: "90d329b0f3d078aace9b29ffc3094653"
server_key_version: "encryptionkey@0"
I20260620 21:58:47.020161 9061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:58:47.020889 9076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:47.021059 9061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:47.021168 9061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "b4f5eb72922b40c191865b03dc9ce6c7"
format_stamp: "Formatted at 2026-06-20 21:58:47 on dist-test-slave-gl2g"
server_key: "5838199c7ee1efffed7eaecb7f81facc"
server_key_iv: "90d329b0f3d078aace9b29ffc3094653"
server_key_version: "encryptionkey@0"
I20260620 21:58:47.021243 9061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:47.030203 9061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:47.032202 9061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:47.032369 9061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:47.032943 9061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:47.033003 9061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:47.033039 9061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:47.033066 9061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:58:47 dist-test-slave-gl2g krb5kdc[8804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992727, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:47.040467 9061 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:36195
I20260620 21:58:47.040611 9191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:36195 every 8 connection(s)
I20260620 21:58:47.040884 9061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:58:47.043133 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 9061
I20260620 21:58:47.043234 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:58:47.043397 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.033368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:39703 (local address 127.6.241.190:40053)
0620 21:58:47.033413 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.033415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.033648 (+ 233us) server_negotiation.cc:407] Connection header received
0620 21:58:47.034191 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.034193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.034206 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.034255 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:47.035030 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.035613 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.036673 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.036924 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.037014 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.040033 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:47.040034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:47.040035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:47.040042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:47.041856 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.042252 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.042252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.042253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.042304 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.042549 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.042550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.042550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.042774 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:47.042842 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.043149 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.043277 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 21:58:47.043432 7110 external_mini_cluster.cc:1428] Setting key 721233b654cbc5d5c75484e155abd0e6
I20260620 21:58:47.043525 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.033369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40053 (local address 127.6.241.130:39703)
0620 21:58:47.033526 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.033561 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.034079 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.034332 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.034335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.034728 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:47.034927 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.034932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.035730 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.035731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.036572 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.036574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.037024 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.037026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.037194 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.037851 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:47.037866 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:47.039927 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:47.041942 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.041947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.041949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.042181 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.042362 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.042363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.042363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.042460 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.042849 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:47.042855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:47.042993 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.043097 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.043281 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":95,"threads_started":1}
I20260620 21:58:47.043993 9192 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40053
I20260620 21:58:47.045717 9192 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:47.045959 9192 heartbeater.cc:511] Master 127.6.241.190:40053 requested a full tablet report, sending...
I20260620 21:58:47.046358 8848 ts_manager.cc:194] Registered new tserver with Master: b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130:36195)
I20260620 21:58:47.046873 8848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:39703
I20260620 21:58:47.051937 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.045108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40053 (local address 127.6.241.130:54159)
0620 21:58:47.045316 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.045327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.045392 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.045510 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.045511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.045739 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:47.045811 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.045814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.046844 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.046845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.047620 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.047622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.048119 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.048120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.048173 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.048687 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:47.048691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:47.048996 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:47.050810 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.050810 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.050811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.050980 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.051197 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.051197 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.051197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.051249 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.051633 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:47.051634 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:47.051677 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.051770 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.051858 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
I20260620 21:58:47.052017 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.045200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:54159 (local address 127.6.241.190:40053)
0620 21:58:47.045258 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.045260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.045394 (+ 134us) server_negotiation.cc:407] Connection header received
0620 21:58:47.045449 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.045449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.045458 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.045523 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:47.045930 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.046741 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.047713 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.048043 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.048123 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.049088 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:47.049089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:47.049090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:47.049099 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:47.050730 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.051062 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.051063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.051063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.051127 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.051306 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.051307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.051308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.051516 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:47.051591 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.051818 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.051903 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:58:47.069685 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40053
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:42167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:47.161996 9201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:47.162222 9201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:47.162252 9201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:47.163501 9201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:47.163552 9201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:47.163574 9201 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:58:47.163585 9201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:58:47.163604 9201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:58:47.165071 9201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:42167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.241.190:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:47.165277 9201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:47.165467 9201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:47.171008 9206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:47.171011 9209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:47.171011 9207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:47.171314 9201 server_base.cc:1034] running on GCE node
I20260620 21:58:47.171571 9201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:47.171851 9201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:47.173022 9201 hybrid_clock.cc:648] HybridClock initialized: now 1781992727173016 us; error 38 us; skew 500 ppm
Jun 20 21:58:47 dist-test-slave-gl2g krb5kdc[8804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992727, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:47.175825 9201 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:58:47.176301 9201 webserver.cc:460] Webserver started at http://127.6.241.131:36179/ using document root <none> and password file <none>
I20260620 21:58:47.176548 9201 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:47.176616 9201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:47.176770 9201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:47.177953 9201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "71a392e94dec40598081302d21f8f447"
format_stamp: "Formatted at 2026-06-20 21:58:47 on dist-test-slave-gl2g"
server_key: "39d7801baabb7e347196e9e6d29d4d05"
server_key_iv: "b2a4c81568944c647782d04d0be9f3c5"
server_key_version: "encryptionkey@0"
I20260620 21:58:47.178347 9201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "71a392e94dec40598081302d21f8f447"
format_stamp: "Formatted at 2026-06-20 21:58:47 on dist-test-slave-gl2g"
server_key: "39d7801baabb7e347196e9e6d29d4d05"
server_key_iv: "b2a4c81568944c647782d04d0be9f3c5"
server_key_version: "encryptionkey@0"
I20260620 21:58:47.179857 9201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:58:47.180673 9216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:47.180891 9201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:47.181001 9201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "71a392e94dec40598081302d21f8f447"
format_stamp: "Formatted at 2026-06-20 21:58:47 on dist-test-slave-gl2g"
server_key: "39d7801baabb7e347196e9e6d29d4d05"
server_key_iv: "b2a4c81568944c647782d04d0be9f3c5"
server_key_version: "encryptionkey@0"
I20260620 21:58:47.181078 9201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:47.190907 9201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:47.193377 9201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:47.193547 9201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:47.194087 9201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:47.194165 9201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:47.194209 9201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:47.194257 9201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:47.201836 9201 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:45189
Jun 20 21:58:47 dist-test-slave-gl2g krb5kdc[8804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992727, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:47.201932 9331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:45189 every 8 connection(s)
I20260620 21:58:47.202232 9201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:58:47.204998 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.194818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:41705 (local address 127.6.241.190:40053)
0620 21:58:47.194865 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.194867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.195113 (+ 246us) server_negotiation.cc:407] Connection header received
0620 21:58:47.195727 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.195729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.195742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.195780 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:47.196470 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.197300 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.199031 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.199363 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.199460 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.202405 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:47.202407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:47.202408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:47.202418 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:47.203789 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.204175 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.204175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.204176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.204242 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.204458 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.204458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.204458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.204569 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:47.204621 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.204862 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.204931 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 21:58:47.205065 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 9201
I20260620 21:58:47.205158 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:58:47.205315 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.194780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40053 (local address 127.6.241.131:41705)
0620 21:58:47.194984 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.195023 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.195605 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.195865 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.195869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.196168 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:47.196368 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.196372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.197386 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.197387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.198926 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.198929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.199492 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.199494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.199728 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.200404 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:47.200415 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:47.202288 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:47.203865 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.203871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.203873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.204102 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.204300 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.204300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.204301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.204398 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.204630 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:47.204634 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:47.204751 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.204833 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.205034 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
I20260620 21:58:47.205350 7110 external_mini_cluster.cc:1428] Setting key 13fdaa318091541e5bbcc3ccf8b7672f
I20260620 21:58:47.205816 9332 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40053
I20260620 21:58:47.205895 9332 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:47.206089 9332 heartbeater.cc:511] Master 127.6.241.190:40053 requested a full tablet report, sending...
I20260620 21:58:47.206789 8848 ts_manager.cc:194] Registered new tserver with Master: 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189)
I20260620 21:58:47.207211 8848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:41705
I20260620 21:58:47.212431 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.207348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:40603 (local address 127.6.241.190:40053)
0620 21:58:47.207385 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.207387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.207395 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:58:47.207414 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.207414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.207424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.207480 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:47.207902 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.208480 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.209193 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.209383 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.209438 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.210268 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:47.210269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:47.210269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:47.210275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:47.211540 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.211853 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.211853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.211854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.211902 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.212058 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.212059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.212059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.212155 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:47.212216 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.212310 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.212364 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 21:58:47.212440 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.206875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40053 (local address 127.6.241.131:40603)
0620 21:58:47.206998 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.207008 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.207052 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.207500 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.207501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.207750 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:47.207815 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.207819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.208592 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.208593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.209113 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.209114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.209449 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.209450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.209504 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.209900 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:47.209905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:47.210193 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:47.211604 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.211605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.211605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.211789 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.211957 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.211957 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.211958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.212007 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.212208 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:47.212208 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:47.212245 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.212294 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.212373 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260620 21:58:47.217082 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:58:47 dist-test-slave-gl2g krb5kdc[8804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:58:47 dist-test-slave-gl2g krb5kdc[8804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:47.239970 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.233201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.6.241.190:40053)
0620 21:58:47.233269 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.233271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.233280 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:58:47.233305 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.233306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.233315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.233369 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:47.233945 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.234457 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.235252 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.235436 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.235494 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.237704 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:47.237705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:47.237706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:47.237713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:47.239011 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.239336 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.239336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.239337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.239391 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.239552 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.239552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.239553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.239664 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:47.239744 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.239837 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.239900 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 21:58:47.241763 8848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:47.243522 8848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:47.250242 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.246103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.6.241.130:36195)
0620 21:58:47.246247 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.246252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.246522 (+ 270us) server_negotiation.cc:407] Connection header received
0620 21:58:47.246584 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.246586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.246603 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.246728 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:47.246901 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.247796 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.249073 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.249835 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.249906 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.249969 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.250099 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.250136 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260620 21:58:47.250229 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.246007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:36195 (local address 127.0.0.1:46366)
0620 21:58:47.246283 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.246327 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.246433 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.246756 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.246759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.246762 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:47.246832 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.246835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.247873 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.247874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.248983 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.248985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.249905 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.249906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.249987 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.250000 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.250070 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.250134 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":61,"threads_started":1}
I20260620 21:58:47.250475 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.246015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:45189 (local address 127.0.0.1:50738)
0620 21:58:47.246683 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.246692 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.246737 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.246996 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.246997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.246999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:47.247047 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.247049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.248428 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.248429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.249371 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.249372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.250236 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.250237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.250302 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.250306 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.250353 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.250397 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":45,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260620 21:58:47.250833 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.246263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.6.241.131:45189)
0620 21:58:47.246444 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.246449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.246795 (+ 346us) server_negotiation.cc:407] Connection header received
0620 21:58:47.246840 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.246842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.246858 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.246928 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:47.247121 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.248329 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.249452 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.250071 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.250140 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.250203 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.250680 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.250726 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260620 21:58:47.251444 9126 tablet_service.cc:1467] Processing CreateTablet for tablet 6ece7a2b97fc4be9bc5a818cc35b0639 (DEFAULT_TABLE table=test-table [id=e47f0b74ed5d4557a420bb5678ee5c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:47.251446 9266 tablet_service.cc:1467] Processing CreateTablet for tablet 6ece7a2b97fc4be9bc5a818cc35b0639 (DEFAULT_TABLE table=test-table [id=e47f0b74ed5d4557a420bb5678ee5c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:47.251832 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ece7a2b97fc4be9bc5a818cc35b0639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:47.251832 9266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ece7a2b97fc4be9bc5a818cc35b0639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:47.254876 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.246015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:41151 (local address 127.0.0.1:52230)
0620 21:58:47.246521 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.246529 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.246581 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.247337 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.247338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.247340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:47.247376 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.247378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.248580 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.248581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.249726 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.249728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.250790 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.250791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.250857 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.250861 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.254701 (+ 3840us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.254760 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":106,"threads_started":1}
I20260620 21:58:47.255254 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.246297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.6.241.129:41151)
0620 21:58:47.246472 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.246482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.246623 (+ 141us) server_negotiation.cc:407] Connection header received
0620 21:58:47.247199 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.247202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.247221 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.247329 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:47.247437 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.248497 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.249838 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.250629 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.250699 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.250754 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.255075 (+ 4321us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.255134 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260620 21:58:47.255471 9355 tablet_bootstrap.cc:492] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: Bootstrap starting.
I20260620 21:58:47.255601 9356 tablet_bootstrap.cc:492] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: Bootstrap starting.
I20260620 21:58:47.256008 8986 tablet_service.cc:1467] Processing CreateTablet for tablet 6ece7a2b97fc4be9bc5a818cc35b0639 (DEFAULT_TABLE table=test-table [id=e47f0b74ed5d4557a420bb5678ee5c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:47.256335 8986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ece7a2b97fc4be9bc5a818cc35b0639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:47.256343 9356 tablet_bootstrap.cc:654] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:47.256554 9355 tablet_bootstrap.cc:654] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:47.256690 9356 log.cc:824] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:47.256891 9355 log.cc:824] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:47.257534 9356 tablet_bootstrap.cc:492] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: No bootstrap required, opened a new log
I20260620 21:58:47.257611 9356 ts_tablet_manager.cc:1397] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:47.257774 9355 tablet_bootstrap.cc:492] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: No bootstrap required, opened a new log
I20260620 21:58:47.257854 9355 ts_tablet_manager.cc:1397] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:58:47.259351 9355 raft_consensus.cc:348] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.259519 9355 raft_consensus.cc:374] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:47.259608 9355 raft_consensus.cc:729] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a392e94dec40598081302d21f8f447, State: Initialized, Role: FOLLOWER
I20260620 21:58:47.259737 9355 consensus_queue.cc:260] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.259774 9356 raft_consensus.cc:348] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.259948 9356 raft_consensus.cc:374] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:47.259997 9356 raft_consensus.cc:729] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f5eb72922b40c191865b03dc9ce6c7, State: Initialized, Role: FOLLOWER
I20260620 21:58:47.260126 9355 ts_tablet_manager.cc:1428] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:58:47.260129 9356 consensus_queue.cc:260] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.260252 9332 heartbeater.cc:503] Master 127.6.241.190:40053 was elected leader, sending a full tablet report...
I20260620 21:58:47.260416 9356 ts_tablet_manager.cc:1428] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 21:58:47.260471 9192 heartbeater.cc:503] Master 127.6.241.190:40053 was elected leader, sending a full tablet report...
I20260620 21:58:47.260757 9361 tablet_bootstrap.cc:492] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: Bootstrap starting.
I20260620 21:58:47.261449 9361 tablet_bootstrap.cc:654] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:47.261826 9361 log.cc:824] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:47.262666 9361 tablet_bootstrap.cc:492] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: No bootstrap required, opened a new log
I20260620 21:58:47.262743 9361 ts_tablet_manager.cc:1397] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:58:47.264091 9361 raft_consensus.cc:348] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.264228 9361 raft_consensus.cc:374] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:47.264252 9361 raft_consensus.cc:729] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b049c8424a7346189c4e551d2f4abadd, State: Initialized, Role: FOLLOWER
I20260620 21:58:47.264335 9361 consensus_queue.cc:260] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.264535 9361 ts_tablet_manager.cc:1428] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:58:47.264607 9052 heartbeater.cc:503] Master 127.6.241.190:40053 was elected leader, sending a full tablet report...
W20260620 21:58:47.291617 9193 tablet.cc:2369] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:58:47.382431 9053 tablet.cc:2369] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:58:47.452880 9333 tablet.cc:2369] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:58:47.454567 9360 raft_consensus.cc:482] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:47.454746 9360 raft_consensus.cc:504] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.455127 9360 leader_election.cc:288] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151), 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189)
I20260620 21:58:47.458282 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.455357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:41151 (local address 127.6.241.130:36333)
0620 21:58:47.455447 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.455474 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.455565 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.455756 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.455757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.455760 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:47.455842 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.455844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.456573 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.456574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.457419 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.457422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.457982 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.457983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.458071 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.458074 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.458151 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.458190 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":30}
I20260620 21:58:47.458267 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.455546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:38153 (local address 127.6.241.131:45189)
0620 21:58:47.455593 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.455595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.455719 (+ 124us) server_negotiation.cc:407] Connection header received
0620 21:58:47.455784 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.455785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.455794 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.455866 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:47.456049 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.456570 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.457525 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.457894 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.457981 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.457991 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.458142 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.458187 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:58:47.458281 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.455442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:36333 (local address 127.6.241.129:41151)
0620 21:58:47.455519 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.455521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.455546 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:58:47.455660 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.455661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.455672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.455726 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:47.455919 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.456477 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.457532 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.457893 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.457982 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.457991 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.458166 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.458212 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:58:47.458299 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.455357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:45189 (local address 127.6.241.130:38153)
0620 21:58:47.455639 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.455649 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.455728 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.455895 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.455896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.455898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:47.455977 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.455978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.456657 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.456658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.457416 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.457419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.457982 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.457983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.458067 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.458071 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.458115 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.458179 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":82,"thread_start_us":190,"threads_started":1}
I20260620 21:58:47.458714 9006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ece7a2b97fc4be9bc5a818cc35b0639" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b049c8424a7346189c4e551d2f4abadd" is_pre_election: true
I20260620 21:58:47.458902 9006 raft_consensus.cc:2454] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 0.
I20260620 21:58:47.458894 9286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ece7a2b97fc4be9bc5a818cc35b0639" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a392e94dec40598081302d21f8f447" is_pre_election: true
I20260620 21:58:47.459021 9286 raft_consensus.cc:2454] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 0.
I20260620 21:58:47.459214 9080 leader_election.cc:302] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b049c8424a7346189c4e551d2f4abadd, b4f5eb72922b40c191865b03dc9ce6c7; no voters:
I20260620 21:58:47.459368 9360 raft_consensus.cc:2781] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:47.459455 9360 raft_consensus.cc:482] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:47.459491 9360 raft_consensus.cc:3037] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:47.460619 9360 raft_consensus.cc:504] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.460786 9360 leader_election.cc:288] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 election: Requested vote from peers b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151), 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189)
I20260620 21:58:47.460965 9006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ece7a2b97fc4be9bc5a818cc35b0639" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b049c8424a7346189c4e551d2f4abadd"
I20260620 21:58:47.460985 9286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ece7a2b97fc4be9bc5a818cc35b0639" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a392e94dec40598081302d21f8f447"
I20260620 21:58:47.461074 9006 raft_consensus.cc:3037] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:47.461150 9286 raft_consensus.cc:3037] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:47.461803 9006 raft_consensus.cc:2454] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 1.
I20260620 21:58:47.461889 9286 raft_consensus.cc:2454] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 1.
I20260620 21:58:47.462037 9080 leader_election.cc:302] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b049c8424a7346189c4e551d2f4abadd, b4f5eb72922b40c191865b03dc9ce6c7; no voters:
I20260620 21:58:47.462133 9360 raft_consensus.cc:2781] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:47.462322 9360 raft_consensus.cc:686] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 LEADER]: Becoming Leader. State: Replica: b4f5eb72922b40c191865b03dc9ce6c7, State: Running, Role: LEADER
I20260620 21:58:47.462421 9360 consensus_queue.cc:237] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.463348 8847 catalog_manager.cc:5495] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130). New cstate: current_term: 1 leader_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } health_report { overall_health: UNKNOWN } } }
Jun 20 21:58:47 dist-test-slave-gl2g krb5kdc[8804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992726, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:47.502913 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.495429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40053 (local address 127.0.0.1:46992)
0620 21:58:47.495614 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.495625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.495698 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.495850 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.495850 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.496162 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:47.496229 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.496231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.496919 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.496920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.497468 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.497469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.497928 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.497929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.497983 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.498658 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:47.498663 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:47.500247 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:47.501874 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.501881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.501884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.502107 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.502322 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:47.502323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:47.502323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:47.502382 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:47.502570 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:47.502577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:47.502653 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.502709 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.502814 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260620 21:58:47.502957 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.495510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.6.241.190:40053)
0620 21:58:47.495637 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.495639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.495679 (+ 40us) server_negotiation.cc:407] Connection header received
0620 21:58:47.495761 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.495762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.495771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.495830 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:47.496333 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.496841 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.497542 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.497854 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.497911 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.500343 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:47.500345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:47.500345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:47.500352 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:47.501792 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.502208 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.502209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.502209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.502262 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:47.502422 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:47.502423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:47.502423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:47.502521 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:47.502565 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.502735 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.502860 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":96}
I20260620 21:58:47.505231 8847 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:46992:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:58:47.505477 8847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:47.507161 8986 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcb7aa3b226438285dc3c5722ea747c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d72e23297df241048bb406951cb6ade2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:58:47.507249 9126 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcb7aa3b226438285dc3c5722ea747c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d72e23297df241048bb406951cb6ade2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:58:47.507401 8986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcb7aa3b226438285dc3c5722ea747c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:47.507413 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcb7aa3b226438285dc3c5722ea747c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:47.508689 9361 tablet_bootstrap.cc:492] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: Bootstrap starting.
I20260620 21:58:47.508859 9356 tablet_bootstrap.cc:492] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: Bootstrap starting.
I20260620 21:58:47.509123 9361 tablet_bootstrap.cc:654] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:47.509265 9356 tablet_bootstrap.cc:654] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:47.509464 9266 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcb7aa3b226438285dc3c5722ea747c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d72e23297df241048bb406951cb6ade2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:58:47.509639 9266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcb7aa3b226438285dc3c5722ea747c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:47.510768 9355 tablet_bootstrap.cc:492] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: Bootstrap starting.
I20260620 21:58:47.511014 9361 tablet_bootstrap.cc:492] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: No bootstrap required, opened a new log
I20260620 21:58:47.511078 9361 ts_tablet_manager.cc:1397] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:47.511250 9361 raft_consensus.cc:348] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.511358 9361 raft_consensus.cc:374] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:47.511358 9355 tablet_bootstrap.cc:654] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:47.511381 9361 raft_consensus.cc:729] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b049c8424a7346189c4e551d2f4abadd, State: Initialized, Role: FOLLOWER
I20260620 21:58:47.511443 9361 consensus_queue.cc:260] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.511580 9361 ts_tablet_manager.cc:1428] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:47.511760 9363 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.511864 9363 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:47.512162 9356 tablet_bootstrap.cc:492] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: No bootstrap required, opened a new log
I20260620 21:58:47.512219 9356 ts_tablet_manager.cc:1397] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 21:58:47.512292 9355 tablet_bootstrap.cc:492] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: No bootstrap required, opened a new log
I20260620 21:58:47.512343 9355 ts_tablet_manager.cc:1397] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:47.512384 9356 raft_consensus.cc:348] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.512466 9356 raft_consensus.cc:374] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:47.512487 9356 raft_consensus.cc:729] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f5eb72922b40c191865b03dc9ce6c7, State: Initialized, Role: FOLLOWER
I20260620 21:58:47.512518 9355 raft_consensus.cc:348] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.512554 9356 consensus_queue.cc:260] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.512590 9355 raft_consensus.cc:374] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:47.512636 9356 ts_tablet_manager.cc:1428] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:47.512653 9355 raft_consensus.cc:729] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a392e94dec40598081302d21f8f447, State: Initialized, Role: FOLLOWER
I20260620 21:58:47.512715 9355 consensus_queue.cc:260] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.512811 9355 ts_tablet_manager.cc:1428] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:47.512778 9360 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.512854 9360 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:47.513016 9359 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.513104 9359 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:47.637085 9363 raft_consensus.cc:482] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:47.637202 9363 raft_consensus.cc:504] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.637542 9363 leader_election.cc:288] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130:36195), 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189)
I20260620 21:58:47.640926 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.637767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:45189 (local address 127.6.241.129:37453)
0620 21:58:47.637909 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.637921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.638038 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.638219 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.638220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.638222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:47.638301 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.638303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.639240 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.639241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.640048 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.640050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.640644 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.640645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.640712 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.640716 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.640792 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.640834 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260620 21:58:47.640913 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.637853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:37453 (local address 127.6.241.131:45189)
0620 21:58:47.637901 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.637904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.638014 (+ 110us) server_negotiation.cc:407] Connection header received
0620 21:58:47.638112 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.638113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.638123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.638198 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:47.638423 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.639148 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.640152 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.640517 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.640603 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.640612 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.640800 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.640844 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:58:47.641202 9286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcb7aa3b226438285dc3c5722ea747c" candidate_uuid: "b049c8424a7346189c4e551d2f4abadd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a392e94dec40598081302d21f8f447" is_pre_election: true
I20260620 21:58:47.641276 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.637767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:36195 (local address 127.6.241.129:36843)
0620 21:58:47.637914 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.637925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.637987 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.638164 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.638165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.638167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:47.638260 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.638262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.639273 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.639274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.640052 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.640054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.640885 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.640886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.640934 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.640938 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.641121 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.641169 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260620 21:58:47.641324 9286 raft_consensus.cc:2454] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b049c8424a7346189c4e551d2f4abadd in term 0.
I20260620 21:58:47.641469 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.637910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:36843 (local address 127.6.241.130:36195)
0620 21:58:47.637960 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.637962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.637973 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:58:47.638058 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.638059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.638074 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.638159 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:47.638339 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.639148 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.640158 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.640522 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.640614 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.640622 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.641337 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.641387 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 21:58:47.641532 8939 leader_election.cc:302] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a392e94dec40598081302d21f8f447, b049c8424a7346189c4e551d2f4abadd; no voters:
I20260620 21:58:47.641738 9363 raft_consensus.cc:2781] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:47.641775 9363 raft_consensus.cc:482] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:47.641789 9363 raft_consensus.cc:3037] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:47.641772 9146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcb7aa3b226438285dc3c5722ea747c" candidate_uuid: "b049c8424a7346189c4e551d2f4abadd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" is_pre_election: true
I20260620 21:58:47.641942 9146 raft_consensus.cc:2454] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b049c8424a7346189c4e551d2f4abadd in term 0.
I20260620 21:58:47.642647 9363 raft_consensus.cc:504] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.642805 9363 leader_election.cc:288] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [CANDIDATE]: Term 1 election: Requested vote from peers b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130:36195), 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189)
I20260620 21:58:47.642974 9146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcb7aa3b226438285dc3c5722ea747c" candidate_uuid: "b049c8424a7346189c4e551d2f4abadd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f5eb72922b40c191865b03dc9ce6c7"
I20260620 21:58:47.642997 9286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcb7aa3b226438285dc3c5722ea747c" candidate_uuid: "b049c8424a7346189c4e551d2f4abadd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a392e94dec40598081302d21f8f447"
I20260620 21:58:47.643054 9146 raft_consensus.cc:3037] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:47.643074 9286 raft_consensus.cc:3037] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:47.643731 9146 raft_consensus.cc:2454] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b049c8424a7346189c4e551d2f4abadd in term 1.
I20260620 21:58:47.643733 9286 raft_consensus.cc:2454] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b049c8424a7346189c4e551d2f4abadd in term 1.
I20260620 21:58:47.643903 8939 leader_election.cc:302] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a392e94dec40598081302d21f8f447, b049c8424a7346189c4e551d2f4abadd; no voters:
I20260620 21:58:47.644187 9363 raft_consensus.cc:2781] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:47.644312 9363 raft_consensus.cc:686] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [term 1 LEADER]: Becoming Leader. State: Replica: b049c8424a7346189c4e551d2f4abadd, State: Running, Role: LEADER
I20260620 21:58:47.644428 9363 consensus_queue.cc:237] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.644912 9378 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: TxnStatusTablet state changed. Reason: New leader b049c8424a7346189c4e551d2f4abadd. Latest consensus state: current_term: 1 leader_uuid: "b049c8424a7346189c4e551d2f4abadd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.645005 9378 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:58:47.645236 9380 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:58:47.645344 9380 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:58:47.645344 8849 catalog_manager.cc:5495] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd reported cstate change: term changed from 0 to 1, leader changed from <none> to b049c8424a7346189c4e551d2f4abadd (127.6.241.129). New cstate: current_term: 1 leader_uuid: "b049c8424a7346189c4e551d2f4abadd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } health_report { overall_health: UNKNOWN } } }
I20260620 21:58:47.662134 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.659346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:41151 (local address 127.0.0.1:52232)
0620 21:58:47.659449 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.659459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.659523 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.659726 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.659727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.659728 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:58:47.659786 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.659788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.660577 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.660578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.661166 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.661168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.661553 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.661555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.661631 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.661647 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:58:47.661947 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:58:47.661957 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.662009 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.662064 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 21:58:47.662132 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.659449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.6.241.129:41151)
0620 21:58:47.659510 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.659513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.659523 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:58:47.659614 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.659615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.659636 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.659722 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:47.659868 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.660471 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.661257 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.661454 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.661544 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.661761 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:47.661872 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:47.661961 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.662015 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.662036 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 21:58:47.664126 9286 raft_consensus.cc:1261] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Refusing update from remote peer b049c8424a7346189c4e551d2f4abadd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:47.664467 9146 raft_consensus.cc:1261] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Refusing update from remote peer b049c8424a7346189c4e551d2f4abadd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:47.664841 9378 consensus_queue.cc:1035] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:47.665005 9363 consensus_queue.cc:1035] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:47.666249 9359 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: TxnStatusTablet state changed. Reason: New leader b049c8424a7346189c4e551d2f4abadd. Latest consensus state: current_term: 1 leader_uuid: "b049c8424a7346189c4e551d2f4abadd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.666384 9359 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:47.666813 9360 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: TxnStatusTablet state changed. Reason: New leader b049c8424a7346189c4e551d2f4abadd. Latest consensus state: current_term: 1 leader_uuid: "b049c8424a7346189c4e551d2f4abadd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.666925 9360 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:47.667894 9363 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b049c8424a7346189c4e551d2f4abadd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.667996 9363 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:58:47.668054 9360 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b049c8424a7346189c4e551d2f4abadd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.668167 9360 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P b4f5eb72922b40c191865b03dc9ce6c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:47.668314 9383 mvcc.cc:205] Tried to move back new op lower bound from 7299042212509552640 to 7299042212432142336. Current Snapshot: MvccSnapshot[applied={T|T < 7299042212509552640}]
I20260620 21:58:47.668551 9378 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b049c8424a7346189c4e551d2f4abadd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.668624 9378 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P b049c8424a7346189c4e551d2f4abadd: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:58:47.668613 9384 mvcc.cc:205] Tried to move back new op lower bound from 7299042212509552640 to 7299042212432142336. Current Snapshot: MvccSnapshot[applied={T|T < 7299042212509552640}]
I20260620 21:58:47.669190 9359 tablet_replica.cc:439] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b049c8424a7346189c4e551d2f4abadd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } } }
I20260620 21:58:47.669291 9359 tablet_replica.cc:442] T 1fcb7aa3b226438285dc3c5722ea747c P 71a392e94dec40598081302d21f8f447: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:58:47.669420 9385 mvcc.cc:205] Tried to move back new op lower bound from 7299042212509552640 to 7299042212432142336. Current Snapshot: MvccSnapshot[applied={T|T < 7299042212509552640}]
I20260620 21:58:47.673913 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.671490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.6.241.130:36195)
0620 21:58:47.671547 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.671549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.671556 (+ 7us) server_negotiation.cc:407] Connection header received
0620 21:58:47.671586 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.671587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.671611 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.671690 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:47.671802 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.672296 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.673105 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.673322 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.673401 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.673549 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:47.673667 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:47.673765 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.673790 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.673811 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 21:58:47.679487 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.676693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:47213 (local address 127.6.241.129:41151)
0620 21:58:47.676740 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.676742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.676837 (+ 95us) server_negotiation.cc:407] Connection header received
0620 21:58:47.676907 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.676908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.676917 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.676962 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:47.677081 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.677529 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.678734 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.678931 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.678998 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.679237 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:47.679272 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:47.679327 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.679402 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.679423 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:58:47.679548 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.676588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:41151 (local address 127.6.241.130:47213)
0620 21:58:47.676778 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.676790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.676853 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.676966 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.676967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.676968 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:58:47.677035 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.677037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.677604 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.677606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.678577 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.678580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.679014 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.679015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.679087 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.679106 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:58:47.679345 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:58:47.679347 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.679400 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.679451 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260620 21:58:47.682483 9286 raft_consensus.cc:1261] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Refusing update from remote peer b4f5eb72922b40c191865b03dc9ce6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:47.682565 9006 raft_consensus.cc:1261] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Refusing update from remote peer b4f5eb72922b40c191865b03dc9ce6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:47.682809 9360 consensus_queue.cc:1035] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:47.682978 9360 consensus_queue.cc:1035] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:47.693356 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.690312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:57725 (local address 127.6.241.130:36195)
0620 21:58:47.690352 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:47.690354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:47.690365 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:58:47.690435 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:47.690436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:47.690446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:47.690506 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:47.690747 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.691618 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.692429 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.692735 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.692809 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.693045 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:47.693084 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:47.693168 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:47.693240 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 21:58:47.693267 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 21:58:47.693396 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:47.690125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:36195 (local address 127.6.241.129:57725)
0620 21:58:47.690307 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:47.690321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:47.690380 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:47.690525 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:47.690526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:47.690528 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:58:47.690644 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.690647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.691701 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.691702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.692343 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:47.692346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:47.692835 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:47.692836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:47.692928 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:47.692948 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:58:47.693168 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:58:47.693170 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:58:47.693227 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 21:58:47.693284 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260620 21:58:47.704362 8848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:47.704504 8848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:47.705387 8848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c: Sending DeleteTablet for 3 replicas of tablet 6ece7a2b97fc4be9bc5a818cc35b0639
I20260620 21:58:47.705916 9266 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ece7a2b97fc4be9bc5a818cc35b0639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:47 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:50738
I20260620 21:58:47.706067 8986 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ece7a2b97fc4be9bc5a818cc35b0639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:47 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:52230
I20260620 21:58:47.706282 9403 tablet_replica.cc:330] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: stopping tablet replica
I20260620 21:58:47.706429 9403 raft_consensus.cc:2229] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:47.706526 9126 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ece7a2b97fc4be9bc5a818cc35b0639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:47 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:46366
I20260620 21:58:47.706533 9403 raft_consensus.cc:2258] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:47.706923 8848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:47.707098 8848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:47.707417 9403 ts_tablet_manager.cc:1905] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:47.708536 9266 tablet_service.cc:1467] Processing CreateTablet for tablet 24e5fc2e6cf240619464a5d3eced1842 (DEFAULT_TABLE table=test-table [id=0cbd6ae754534fd1871379b7b70e3e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:47.708674 9266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e5fc2e6cf240619464a5d3eced1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:47.709438 9403 ts_tablet_manager.cc:1918] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:58:47.709583 9403 log.cc:1196] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/6ece7a2b97fc4be9bc5a818cc35b0639
I20260620 21:58:47.709933 9403 ts_tablet_manager.cc:1939] T 6ece7a2b97fc4be9bc5a818cc35b0639 P 71a392e94dec40598081302d21f8f447: Deleting consensus metadata
I20260620 21:58:47.710212 9355 tablet_bootstrap.cc:492] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: Bootstrap starting.
I20260620 21:58:47.710657 9355 tablet_bootstrap.cc:654] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:47.710932 8835 catalog_manager.cc:4841] TS 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189): tablet 6ece7a2b97fc4be9bc5a818cc35b0639 (table test-table [id=e47f0b74ed5d4557a420bb5678ee5c8b]) successfully deleted
I20260620 21:58:47.711278 9355 tablet_bootstrap.cc:492] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: No bootstrap required, opened a new log
I20260620 21:58:47.711326 9355 ts_tablet_manager.cc:1397] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:47.711302 9126 tablet_service.cc:1467] Processing CreateTablet for tablet 24e5fc2e6cf240619464a5d3eced1842 (DEFAULT_TABLE table=test-table [id=0cbd6ae754534fd1871379b7b70e3e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:47.711421 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e5fc2e6cf240619464a5d3eced1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:47.711467 9355 raft_consensus.cc:348] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.711545 9355 raft_consensus.cc:374] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:47.711565 9355 raft_consensus.cc:729] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a392e94dec40598081302d21f8f447, State: Initialized, Role: FOLLOWER
I20260620 21:58:47.711611 9355 consensus_queue.cc:260] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.711695 9355 ts_tablet_manager.cc:1428] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:47.712726 9356 tablet_bootstrap.cc:492] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: Bootstrap starting.
I20260620 21:58:47.713145 9356 tablet_bootstrap.cc:654] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:47.713851 9356 tablet_bootstrap.cc:492] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: No bootstrap required, opened a new log
I20260620 21:58:47.713922 9356 ts_tablet_manager.cc:1397] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:47.714084 8986 tablet_service.cc:1467] Processing CreateTablet for tablet 24e5fc2e6cf240619464a5d3eced1842 (DEFAULT_TABLE table=test-table [id=0cbd6ae754534fd1871379b7b70e3e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:47.714075 9356 raft_consensus.cc:348] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.714161 9356 raft_consensus.cc:374] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:47.714187 9356 raft_consensus.cc:729] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f5eb72922b40c191865b03dc9ce6c7, State: Initialized, Role: FOLLOWER
I20260620 21:58:47.714205 8986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e5fc2e6cf240619464a5d3eced1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:47.714231 9356 consensus_queue.cc:260] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.714334 9356 ts_tablet_manager.cc:1428] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:47.715591 9361 tablet_bootstrap.cc:492] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: Bootstrap starting.
I20260620 21:58:47.715945 9361 tablet_bootstrap.cc:654] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:47.716341 9404 tablet_replica.cc:330] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: stopping tablet replica
I20260620 21:58:47.716490 9405 tablet_replica.cc:330] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: stopping tablet replica
I20260620 21:58:47.716650 9405 raft_consensus.cc:2229] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:47.716683 9361 tablet_bootstrap.cc:492] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: No bootstrap required, opened a new log
I20260620 21:58:47.716774 9361 ts_tablet_manager.cc:1397] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:47.716903 9404 raft_consensus.cc:2229] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:47.717087 9405 raft_consensus.cc:2258] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:47.716917 9361 raft_consensus.cc:348] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.717221 9361 raft_consensus.cc:374] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:47.717250 9361 raft_consensus.cc:729] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b049c8424a7346189c4e551d2f4abadd, State: Initialized, Role: FOLLOWER
I20260620 21:58:47.717296 9361 consensus_queue.cc:260] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.717410 9361 ts_tablet_manager.cc:1428] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:58:47.717768 9405 ts_tablet_manager.cc:1905] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:47.718178 9404 raft_consensus.cc:2258] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:47.718633 9404 ts_tablet_manager.cc:1905] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:47.720335 9404 ts_tablet_manager.cc:1918] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:58:47.720412 9404 log.cc:1196] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/6ece7a2b97fc4be9bc5a818cc35b0639
I20260620 21:58:47.720685 9404 ts_tablet_manager.cc:1939] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b049c8424a7346189c4e551d2f4abadd: Deleting consensus metadata
I20260620 21:58:47.721176 8836 catalog_manager.cc:4841] TS b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151): tablet 6ece7a2b97fc4be9bc5a818cc35b0639 (table test-table [id=e47f0b74ed5d4557a420bb5678ee5c8b]) successfully deleted
I20260620 21:58:47.722779 9405 ts_tablet_manager.cc:1918] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:58:47.722887 9405 log.cc:1196] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/6ece7a2b97fc4be9bc5a818cc35b0639
I20260620 21:58:47.723201 9405 ts_tablet_manager.cc:1939] T 6ece7a2b97fc4be9bc5a818cc35b0639 P b4f5eb72922b40c191865b03dc9ce6c7: Deleting consensus metadata
I20260620 21:58:47.723624 8833 catalog_manager.cc:4841] TS b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130:36195): tablet 6ece7a2b97fc4be9bc5a818cc35b0639 (table test-table [id=e47f0b74ed5d4557a420bb5678ee5c8b]) successfully deleted
I20260620 21:58:47.870438 9401 raft_consensus.cc:482] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:47.870549 9401 raft_consensus.cc:504] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.870826 9401 leader_election.cc:288] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151), 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189)
I20260620 21:58:47.871104 9286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e5fc2e6cf240619464a5d3eced1842" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a392e94dec40598081302d21f8f447" is_pre_election: true
I20260620 21:58:47.871104 9006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e5fc2e6cf240619464a5d3eced1842" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b049c8424a7346189c4e551d2f4abadd" is_pre_election: true
I20260620 21:58:47.871219 9006 raft_consensus.cc:2454] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 0.
I20260620 21:58:47.871227 9286 raft_consensus.cc:2454] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 0.
I20260620 21:58:47.871389 9080 leader_election.cc:302] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b049c8424a7346189c4e551d2f4abadd, b4f5eb72922b40c191865b03dc9ce6c7; no voters:
I20260620 21:58:47.871531 9401 raft_consensus.cc:2781] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:47.871588 9401 raft_consensus.cc:482] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:47.871614 9401 raft_consensus.cc:3037] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:47.872350 9401 raft_consensus.cc:504] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.872529 9401 leader_election.cc:288] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 election: Requested vote from peers b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151), 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189)
I20260620 21:58:47.872694 9286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e5fc2e6cf240619464a5d3eced1842" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a392e94dec40598081302d21f8f447"
I20260620 21:58:47.872807 9286 raft_consensus.cc:3037] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:47.872757 9006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e5fc2e6cf240619464a5d3eced1842" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b049c8424a7346189c4e551d2f4abadd"
I20260620 21:58:47.872884 9006 raft_consensus.cc:3037] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:47.873531 9286 raft_consensus.cc:2454] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 1.
I20260620 21:58:47.873654 9006 raft_consensus.cc:2454] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 1.
I20260620 21:58:47.873769 9079 leader_election.cc:302] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a392e94dec40598081302d21f8f447, b4f5eb72922b40c191865b03dc9ce6c7; no voters:
I20260620 21:58:47.873876 9401 raft_consensus.cc:2781] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:47.873955 9401 raft_consensus.cc:686] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 LEADER]: Becoming Leader. State: Replica: b4f5eb72922b40c191865b03dc9ce6c7, State: Running, Role: LEADER
I20260620 21:58:47.874017 9401 consensus_queue.cc:237] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:47.874724 8849 catalog_manager.cc:5495] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130). New cstate: current_term: 1 leader_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } health_report { overall_health: UNKNOWN } } }
I20260620 21:58:47.904244 9006 raft_consensus.cc:1261] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Refusing update from remote peer b4f5eb72922b40c191865b03dc9ce6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:47.904248 9286 raft_consensus.cc:1261] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Refusing update from remote peer b4f5eb72922b40c191865b03dc9ce6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:47.904472 9401 consensus_queue.cc:1035] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:47.904620 9365 consensus_queue.cc:1035] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:58:47.909623 9384 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 21:58:49.427515 8849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:49.427625 8849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:49.428534 8849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c: Sending DeleteTablet for 3 replicas of tablet 24e5fc2e6cf240619464a5d3eced1842
I20260620 21:58:49.428839 8986 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e5fc2e6cf240619464a5d3eced1842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:49 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:52230
I20260620 21:58:49.428906 9266 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e5fc2e6cf240619464a5d3eced1842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:49 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:50738
I20260620 21:58:49.428918 9126 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e5fc2e6cf240619464a5d3eced1842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:49 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:46366
I20260620 21:58:49.429155 9435 tablet_replica.cc:330] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: stopping tablet replica
I20260620 21:58:49.429225 9435 raft_consensus.cc:2229] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:49.429262 9435 raft_consensus.cc:2258] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:49.429500 9435 ts_tablet_manager.cc:1905] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:49.431538 9435 ts_tablet_manager.cc:1918] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:58:49.431613 9435 log.cc:1196] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/24e5fc2e6cf240619464a5d3eced1842
I20260620 21:58:49.431777 9436 tablet_replica.cc:330] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: stopping tablet replica
I20260620 21:58:49.431847 9435 ts_tablet_manager.cc:1939] T 24e5fc2e6cf240619464a5d3eced1842 P b049c8424a7346189c4e551d2f4abadd: Deleting consensus metadata
I20260620 21:58:49.431850 9436 raft_consensus.cc:2229] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:49.431939 9436 raft_consensus.cc:2258] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:49.432065 9437 tablet_replica.cc:330] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: stopping tablet replica
I20260620 21:58:49.432140 9437 raft_consensus.cc:2229] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:49.432190 9437 raft_consensus.cc:2258] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:49.432200 8836 catalog_manager.cc:4841] TS b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151): tablet 24e5fc2e6cf240619464a5d3eced1842 (table test-table [id=0cbd6ae754534fd1871379b7b70e3e40]) successfully deleted
I20260620 21:58:49.432425 9437 ts_tablet_manager.cc:1905] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:49.432520 9436 ts_tablet_manager.cc:1905] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:49.433629 9437 ts_tablet_manager.cc:1918] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:58:49.433688 9437 log.cc:1196] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/24e5fc2e6cf240619464a5d3eced1842
I20260620 21:58:49.433923 9437 ts_tablet_manager.cc:1939] T 24e5fc2e6cf240619464a5d3eced1842 P 71a392e94dec40598081302d21f8f447: Deleting consensus metadata
I20260620 21:58:49.434248 8835 catalog_manager.cc:4841] TS 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189): tablet 24e5fc2e6cf240619464a5d3eced1842 (table test-table [id=0cbd6ae754534fd1871379b7b70e3e40]) successfully deleted
I20260620 21:58:49.434417 9436 ts_tablet_manager.cc:1918] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:58:49.434482 9436 log.cc:1196] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/24e5fc2e6cf240619464a5d3eced1842
I20260620 21:58:49.434775 9436 ts_tablet_manager.cc:1939] T 24e5fc2e6cf240619464a5d3eced1842 P b4f5eb72922b40c191865b03dc9ce6c7: Deleting consensus metadata
I20260620 21:58:49.435109 8833 catalog_manager.cc:4841] TS b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130:36195): tablet 24e5fc2e6cf240619464a5d3eced1842 (table test-table [id=0cbd6ae754534fd1871379b7b70e3e40]) successfully deleted
Jun 20 21:58:49 dist-test-slave-gl2g krb5kdc[8804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:58:49 dist-test-slave-gl2g krb5kdc[8804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:49.459208 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.446426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.6.241.190:40053)
0620 21:58:49.446646 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:49.446648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:49.446662 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:58:49.446689 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:49.446690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:49.446700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:49.446773 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:49.447335 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.447892 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.448714 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.448942 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.449006 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.456453 (+ 7447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:49.456456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:49.456456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:49.456465 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:49.458040 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:49.458413 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:49.458414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:49.458415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:49.458487 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:49.458737 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:49.458738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:49.458739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:49.458862 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:49.458947 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:49.459054 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 21:58:49.459132 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260620 21:58:49.460945 8849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:49.461102 8849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:49.462535 8986 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0d6539305542e5a95589c1736c824d (DEFAULT_TABLE table=test-table [id=8f6c374569df4c7d98975329a28b1271]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:49.462574 9126 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0d6539305542e5a95589c1736c824d (DEFAULT_TABLE table=test-table [id=8f6c374569df4c7d98975329a28b1271]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:49.462682 9266 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0d6539305542e5a95589c1736c824d (DEFAULT_TABLE table=test-table [id=8f6c374569df4c7d98975329a28b1271]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:49.462735 8986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0d6539305542e5a95589c1736c824d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:49.462755 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0d6539305542e5a95589c1736c824d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:49.462793 9266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0d6539305542e5a95589c1736c824d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:49.463946 9446 tablet_bootstrap.cc:492] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: Bootstrap starting.
I20260620 21:58:49.464151 9447 tablet_bootstrap.cc:492] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: Bootstrap starting.
I20260620 21:58:49.464470 9446 tablet_bootstrap.cc:654] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:49.464545 9447 tablet_bootstrap.cc:654] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:49.464862 9448 tablet_bootstrap.cc:492] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: Bootstrap starting.
I20260620 21:58:49.465147 9446 tablet_bootstrap.cc:492] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: No bootstrap required, opened a new log
I20260620 21:58:49.465207 9446 ts_tablet_manager.cc:1397] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:49.465273 9448 tablet_bootstrap.cc:654] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:49.465337 9446 raft_consensus.cc:348] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.465428 9446 raft_consensus.cc:374] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:49.465456 9446 raft_consensus.cc:729] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b049c8424a7346189c4e551d2f4abadd, State: Initialized, Role: FOLLOWER
I20260620 21:58:49.465521 9446 consensus_queue.cc:260] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.465662 9446 ts_tablet_manager.cc:1428] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:49.465903 9447 tablet_bootstrap.cc:492] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: No bootstrap required, opened a new log
I20260620 21:58:49.465986 9447 ts_tablet_manager.cc:1397] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:49.466102 9447 raft_consensus.cc:348] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.466187 9447 raft_consensus.cc:374] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:49.466208 9447 raft_consensus.cc:729] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f5eb72922b40c191865b03dc9ce6c7, State: Initialized, Role: FOLLOWER
I20260620 21:58:49.466262 9447 consensus_queue.cc:260] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.466361 9447 ts_tablet_manager.cc:1428] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:49.466722 9448 tablet_bootstrap.cc:492] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: No bootstrap required, opened a new log
I20260620 21:58:49.466785 9448 ts_tablet_manager.cc:1397] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:49.466917 9448 raft_consensus.cc:348] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.467023 9448 raft_consensus.cc:374] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:49.467047 9448 raft_consensus.cc:729] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a392e94dec40598081302d21f8f447, State: Initialized, Role: FOLLOWER
I20260620 21:58:49.467098 9448 consensus_queue.cc:260] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.467285 9448 ts_tablet_manager.cc:1428] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:49.712098 9452 raft_consensus.cc:482] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:49.712225 9452 raft_consensus.cc:504] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.712616 9452 leader_election.cc:288] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151), b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130:36195)
I20260620 21:58:49.716969 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.712827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:41151 (local address 127.6.241.131:60513)
0620 21:58:49.713011 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:49.713023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:49.713083 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:49.713291 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:49.713292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:49.713294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:49.713406 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:49.713408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.714400 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.714402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:49.715909 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:49.715913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.716673 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.716673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:49.716734 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.716738 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:49.716785 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 21:58:49.716856 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":87,"thread_start_us":100,"threads_started":1}
I20260620 21:58:49.716974 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.712933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:60513 (local address 127.6.241.129:41151)
0620 21:58:49.713105 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:49.713107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:49.713144 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:58:49.713176 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:49.713176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:49.713187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:49.713266 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:49.713500 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.714298 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.716015 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.716572 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.716657 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.716667 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:49.716835 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 21:58:49.716891 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260620 21:58:49.717113 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.712930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:57717 (local address 127.6.241.130:36195)
0620 21:58:49.713102 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:49.713105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:49.713144 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:58:49.713175 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:49.713176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:49.713187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:49.713265 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:49.713500 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.714268 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.716178 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.716747 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.716835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.716843 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:49.716988 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 21:58:49.717036 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260620 21:58:49.717119 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.712827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:36195 (local address 127.6.241.131:57717)
0620 21:58:49.713010 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:49.713023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:49.713083 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:49.713294 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:49.713294 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:49.713296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:49.713406 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:49.713409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.714400 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.714402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:49.716087 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:49.716089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.716848 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.716849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:49.716907 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.716911 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:58:49.716964 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:58:49.717015 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":68,"thread_start_us":103,"threads_started":1}
I20260620 21:58:49.717371 9146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0d6539305542e5a95589c1736c824d" candidate_uuid: "71a392e94dec40598081302d21f8f447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" is_pre_election: true
I20260620 21:58:49.717487 9146 raft_consensus.cc:2454] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a392e94dec40598081302d21f8f447 in term 0.
I20260620 21:58:49.717511 9006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0d6539305542e5a95589c1736c824d" candidate_uuid: "71a392e94dec40598081302d21f8f447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b049c8424a7346189c4e551d2f4abadd" is_pre_election: true
I20260620 21:58:49.717604 9006 raft_consensus.cc:2454] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a392e94dec40598081302d21f8f447 in term 0.
I20260620 21:58:49.717680 9217 leader_election.cc:302] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a392e94dec40598081302d21f8f447, b4f5eb72922b40c191865b03dc9ce6c7; no voters:
I20260620 21:58:49.717859 9452 raft_consensus.cc:2781] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:49.717916 9452 raft_consensus.cc:482] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:49.717969 9452 raft_consensus.cc:3037] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:49.718748 9452 raft_consensus.cc:504] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.718914 9452 leader_election.cc:288] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [CANDIDATE]: Term 1 election: Requested vote from peers b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151), b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130:36195)
I20260620 21:58:49.719121 9006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0d6539305542e5a95589c1736c824d" candidate_uuid: "71a392e94dec40598081302d21f8f447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b049c8424a7346189c4e551d2f4abadd"
I20260620 21:58:49.719117 9146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0d6539305542e5a95589c1736c824d" candidate_uuid: "71a392e94dec40598081302d21f8f447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f5eb72922b40c191865b03dc9ce6c7"
I20260620 21:58:49.719197 9146 raft_consensus.cc:3037] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:49.719219 9006 raft_consensus.cc:3037] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:49.719811 9146 raft_consensus.cc:2454] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a392e94dec40598081302d21f8f447 in term 1.
I20260620 21:58:49.719815 9006 raft_consensus.cc:2454] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a392e94dec40598081302d21f8f447 in term 1.
I20260620 21:58:49.719972 9217 leader_election.cc:302] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a392e94dec40598081302d21f8f447, b4f5eb72922b40c191865b03dc9ce6c7; no voters:
I20260620 21:58:49.720104 9452 raft_consensus.cc:2781] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:49.720293 9452 raft_consensus.cc:686] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 1 LEADER]: Becoming Leader. State: Replica: 71a392e94dec40598081302d21f8f447, State: Running, Role: LEADER
I20260620 21:58:49.720407 9452 consensus_queue.cc:237] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.721189 8847 catalog_manager.cc:5495] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a392e94dec40598081302d21f8f447 (127.6.241.131). New cstate: current_term: 1 leader_uuid: "71a392e94dec40598081302d21f8f447" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } health_report { overall_health: HEALTHY } } }
I20260620 21:58:49.754437 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.751801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.6.241.131:45189)
0620 21:58:49.751959 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:49.751961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:49.751969 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:58:49.752015 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:49.752015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:49.752036 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:49.752109 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:49.752224 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.752733 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.753615 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.753850 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.753903 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.754073 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:49.754214 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:49.754294 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:49.754329 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 21:58:49.754347 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":112,"threads_started":1}
I20260620 21:58:49.760274 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.757921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:41151 (local address 127.6.241.131:60865)
0620 21:58:49.758076 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:49.758086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:49.758137 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:49.758254 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:49.758254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:49.758256 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:58:49.758305 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:49.758306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.758961 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.758962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:49.759605 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:49.759608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.759923 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.759924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:49.759982 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.759995 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:58:49.760127 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:58:49.760128 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 21:58:49.760158 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 21:58:49.760190 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260620 21:58:49.760281 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.757992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:60865 (local address 127.6.241.129:41151)
0620 21:58:49.758060 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:49.758063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:49.758128 (+ 65us) server_negotiation.cc:407] Connection header received
0620 21:58:49.758196 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:49.758197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:49.758206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:49.758245 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:49.758351 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.758896 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.759677 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.759862 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.759915 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.760064 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:49.760090 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:49.760127 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:49.760192 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 21:58:49.760215 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:58:49.762828 9146 raft_consensus.cc:1261] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Refusing update from remote peer 71a392e94dec40598081302d21f8f447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:49.762828 9006 raft_consensus.cc:1261] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Refusing update from remote peer 71a392e94dec40598081302d21f8f447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:49.763068 9452 consensus_queue.cc:1035] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:49.763257 9457 consensus_queue.cc:1035] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:49.770512 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.768154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:49083 (local address 127.6.241.131:45189)
0620 21:58:49.768199 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:49.768201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:49.768210 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:58:49.768289 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:49.768290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:49.768303 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:49.768356 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:49.768464 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.768940 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.769686 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.769953 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.770039 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.770192 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:49.770227 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:49.770299 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:49.770424 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 21:58:49.770454 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:58:49.770524 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.768018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:45189 (local address 127.6.241.129:49083)
0620 21:58:49.768145 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:49.768157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:49.768209 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:49.768358 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:49.768359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:49.768361 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:58:49.768412 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:49.768414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.769007 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.769007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:49.769585 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:49.769587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.770045 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.770046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:49.770111 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.770117 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:58:49.770352 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:58:49.770354 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:58:49.770404 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 21:58:49.770462 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":86,"threads_started":1}
I20260620 21:58:49.777035 8849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:49.777150 8849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:49.777773 8849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c: Sending DeleteTablet for 3 replicas of tablet 3b0d6539305542e5a95589c1736c824d
I20260620 21:58:49.778002 8986 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0d6539305542e5a95589c1736c824d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:49 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:52230
I20260620 21:58:49.778031 9266 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0d6539305542e5a95589c1736c824d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:49 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:50738
I20260620 21:58:49.778100 9435 tablet_replica.cc:330] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: stopping tablet replica
I20260620 21:58:49.778122 9437 tablet_replica.cc:330] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: stopping tablet replica
I20260620 21:58:49.778105 9126 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0d6539305542e5a95589c1736c824d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:49 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:46366
I20260620 21:58:49.778148 9435 raft_consensus.cc:2229] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:49.778167 9437 raft_consensus.cc:2229] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:49.778187 9435 raft_consensus.cc:2258] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:49.778187 9436 tablet_replica.cc:330] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: stopping tablet replica
I20260620 21:58:49.778228 9436 raft_consensus.cc:2229] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:49.778259 9436 raft_consensus.cc:2258] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:49.778260 9437 raft_consensus.cc:2258] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:49.778630 9437 ts_tablet_manager.cc:1905] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:49.778684 9435 ts_tablet_manager.cc:1905] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:49.780301 9436 ts_tablet_manager.cc:1905] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:49.780359 9437 ts_tablet_manager.cc:1918] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:58:49.780406 9437 log.cc:1196] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/3b0d6539305542e5a95589c1736c824d
I20260620 21:58:49.780624 9437 ts_tablet_manager.cc:1939] T 3b0d6539305542e5a95589c1736c824d P 71a392e94dec40598081302d21f8f447: Deleting consensus metadata
I20260620 21:58:49.780975 8835 catalog_manager.cc:4841] TS 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189): tablet 3b0d6539305542e5a95589c1736c824d (table test-table [id=8f6c374569df4c7d98975329a28b1271]) successfully deleted
I20260620 21:58:49.781654 9436 ts_tablet_manager.cc:1918] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:58:49.781704 9436 log.cc:1196] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/3b0d6539305542e5a95589c1736c824d
I20260620 21:58:49.781934 9436 ts_tablet_manager.cc:1939] T 3b0d6539305542e5a95589c1736c824d P b4f5eb72922b40c191865b03dc9ce6c7: Deleting consensus metadata
I20260620 21:58:49.782232 8849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:49.782408 8849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:49.782684 8833 catalog_manager.cc:4841] TS b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130:36195): tablet 3b0d6539305542e5a95589c1736c824d (table test-table [id=8f6c374569df4c7d98975329a28b1271]) successfully deleted
I20260620 21:58:49.782770 9435 ts_tablet_manager.cc:1918] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:58:49.782825 9435 log.cc:1196] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/3b0d6539305542e5a95589c1736c824d
I20260620 21:58:49.783044 9435 ts_tablet_manager.cc:1939] T 3b0d6539305542e5a95589c1736c824d P b049c8424a7346189c4e551d2f4abadd: Deleting consensus metadata
I20260620 21:58:49.783366 8836 catalog_manager.cc:4841] TS b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151): tablet 3b0d6539305542e5a95589c1736c824d (table test-table [id=8f6c374569df4c7d98975329a28b1271]) successfully deleted
I20260620 21:58:49.784065 9126 tablet_service.cc:1467] Processing CreateTablet for tablet 6a31c07ade9844289468a67613b18ba8 (DEFAULT_TABLE table=test-table [id=40281701be5c43889ebb32b945cad56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:49.784071 9266 tablet_service.cc:1467] Processing CreateTablet for tablet 6a31c07ade9844289468a67613b18ba8 (DEFAULT_TABLE table=test-table [id=40281701be5c43889ebb32b945cad56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:49.784067 8986 tablet_service.cc:1467] Processing CreateTablet for tablet 6a31c07ade9844289468a67613b18ba8 (DEFAULT_TABLE table=test-table [id=40281701be5c43889ebb32b945cad56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:49.784179 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a31c07ade9844289468a67613b18ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:49.784186 8986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a31c07ade9844289468a67613b18ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:49.784191 9266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a31c07ade9844289468a67613b18ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:49.785475 9448 tablet_bootstrap.cc:492] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: Bootstrap starting.
I20260620 21:58:49.785480 9447 tablet_bootstrap.cc:492] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: Bootstrap starting.
I20260620 21:58:49.785559 9446 tablet_bootstrap.cc:492] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd: Bootstrap starting.
I20260620 21:58:49.785821 9447 tablet_bootstrap.cc:654] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:49.785881 9448 tablet_bootstrap.cc:654] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:49.785887 9446 tablet_bootstrap.cc:654] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:49.786585 9446 tablet_bootstrap.cc:492] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd: No bootstrap required, opened a new log
I20260620 21:58:49.786671 9446 ts_tablet_manager.cc:1397] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:49.786769 9448 tablet_bootstrap.cc:492] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: No bootstrap required, opened a new log
I20260620 21:58:49.786805 9448 ts_tablet_manager.cc:1397] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:49.786792 9446 raft_consensus.cc:348] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.786862 9446 raft_consensus.cc:374] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:49.786882 9446 raft_consensus.cc:729] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b049c8424a7346189c4e551d2f4abadd, State: Initialized, Role: FOLLOWER
I20260620 21:58:49.786921 9446 consensus_queue.cc:260] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.786937 9448 raft_consensus.cc:348] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.786985 9448 raft_consensus.cc:374] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:49.786998 9448 raft_consensus.cc:729] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a392e94dec40598081302d21f8f447, State: Initialized, Role: FOLLOWER
I20260620 21:58:49.787002 9446 ts_tablet_manager.cc:1428] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:49.787062 9448 consensus_queue.cc:260] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.787170 9448 ts_tablet_manager.cc:1428] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:49.787555 9447 tablet_bootstrap.cc:492] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: No bootstrap required, opened a new log
I20260620 21:58:49.787609 9447 ts_tablet_manager.cc:1397] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:49.787715 9447 raft_consensus.cc:348] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.787823 9447 raft_consensus.cc:374] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:49.787853 9447 raft_consensus.cc:729] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f5eb72922b40c191865b03dc9ce6c7, State: Initialized, Role: FOLLOWER
I20260620 21:58:49.787896 9447 consensus_queue.cc:260] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.788005 9447 ts_tablet_manager.cc:1428] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:49.851454 9428 raft_consensus.cc:482] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:49.851585 9428 raft_consensus.cc:504] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.851784 9428 leader_election.cc:288] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151), 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189)
I20260620 21:58:49.852052 9006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a31c07ade9844289468a67613b18ba8" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b049c8424a7346189c4e551d2f4abadd" is_pre_election: true
I20260620 21:58:49.852061 9285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a31c07ade9844289468a67613b18ba8" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a392e94dec40598081302d21f8f447" is_pre_election: true
I20260620 21:58:49.852166 9006 raft_consensus.cc:2454] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 0.
I20260620 21:58:49.852166 9285 raft_consensus.cc:2454] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 0.
I20260620 21:58:49.852334 9079 leader_election.cc:302] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a392e94dec40598081302d21f8f447, b4f5eb72922b40c191865b03dc9ce6c7; no voters:
I20260620 21:58:49.852444 9428 raft_consensus.cc:2781] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:49.852504 9428 raft_consensus.cc:482] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:49.852526 9428 raft_consensus.cc:3037] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:49.853492 9428 raft_consensus.cc:504] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.853637 9428 leader_election.cc:288] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 election: Requested vote from peers b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151), 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189)
I20260620 21:58:49.853850 9285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a31c07ade9844289468a67613b18ba8" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a392e94dec40598081302d21f8f447"
I20260620 21:58:49.853864 9006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a31c07ade9844289468a67613b18ba8" candidate_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b049c8424a7346189c4e551d2f4abadd"
I20260620 21:58:49.853940 9285 raft_consensus.cc:3037] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:49.853960 9006 raft_consensus.cc:3037] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:49.854794 9285 raft_consensus.cc:2454] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 1.
I20260620 21:58:49.854794 9006 raft_consensus.cc:2454] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f5eb72922b40c191865b03dc9ce6c7 in term 1.
I20260620 21:58:49.854969 9080 leader_election.cc:302] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b049c8424a7346189c4e551d2f4abadd, b4f5eb72922b40c191865b03dc9ce6c7; no voters:
I20260620 21:58:49.855094 9428 raft_consensus.cc:2781] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:49.855162 9428 raft_consensus.cc:686] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 LEADER]: Becoming Leader. State: Replica: b4f5eb72922b40c191865b03dc9ce6c7, State: Running, Role: LEADER
I20260620 21:58:49.855213 9428 consensus_queue.cc:237] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } }
I20260620 21:58:49.855895 8849 catalog_manager.cc:5495] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f5eb72922b40c191865b03dc9ce6c7 (127.6.241.130). New cstate: current_term: 1 leader_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f5eb72922b40c191865b03dc9ce6c7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 } health_report { overall_health: UNKNOWN } } }
I20260620 21:58:49.865728 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:49.863448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.6.241.130:36195)
0620 21:58:49.863491 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:49.863494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:49.863502 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:58:49.863568 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:49.863569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:49.863580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:49.863635 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:49.863764 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.864251 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.864996 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:49.865216 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:49.865292 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:49.865451 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:49.865485 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:49.865556 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:49.865617 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 21:58:49.865638 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:58:49.868709 9285 raft_consensus.cc:1261] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Refusing update from remote peer b4f5eb72922b40c191865b03dc9ce6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:49.868750 9006 raft_consensus.cc:1261] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Refusing update from remote peer b4f5eb72922b40c191865b03dc9ce6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:49.868927 9428 consensus_queue.cc:1035] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a392e94dec40598081302d21f8f447" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:49.869019 9365 consensus_queue.cc:1035] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b049c8424a7346189c4e551d2f4abadd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 41151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:58:49.873896 9417 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 21:58:51.392436 8849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:51.392570 8849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:51.393450 8849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a36630912734c67a02e32c963f9774c: Sending DeleteTablet for 3 replicas of tablet 6a31c07ade9844289468a67613b18ba8
I20260620 21:58:51.393695 8986 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a31c07ade9844289468a67613b18ba8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:51 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:52230
I20260620 21:58:51.393736 9266 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a31c07ade9844289468a67613b18ba8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:51 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:50738
I20260620 21:58:51.393896 9126 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a31c07ade9844289468a67613b18ba8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:51 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:46366
I20260620 21:58:51.393929 9498 tablet_replica.cc:330] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd: stopping tablet replica
I20260620 21:58:51.393985 9498 raft_consensus.cc:2229] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:51.394024 9498 raft_consensus.cc:2258] T 6a31c07ade9844289468a67613b18ba8 P b049c8424a7346189c4e551d2f4abadd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:51.394219 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 8921
I20260620 21:58:51.398785 9500 tablet_replica.cc:330] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: stopping tablet replica
I20260620 21:58:51.398800 9499 tablet_replica.cc:330] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: stopping tablet replica
I20260620 21:58:51.398898 9500 raft_consensus.cc:2229] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:51.398898 9499 raft_consensus.cc:2229] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:51.398950 9499 raft_consensus.cc:2258] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:51.399010 9500 raft_consensus.cc:2258] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:51.399318 9500 ts_tablet_manager.cc:1905] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:51.401198 9500 ts_tablet_manager.cc:1918] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:58:51.401274 9500 log.cc:1196] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/6a31c07ade9844289468a67613b18ba8
I20260620 21:58:51.401547 9500 ts_tablet_manager.cc:1939] T 6a31c07ade9844289468a67613b18ba8 P b4f5eb72922b40c191865b03dc9ce6c7: Deleting consensus metadata
W20260620 21:58:51.401712 8836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:41151 (error 108)
I20260620 21:58:51.401769 9499 ts_tablet_manager.cc:1905] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:51.401852 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 9061
I20260620 21:58:51.403898 9499 ts_tablet_manager.cc:1918] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:58:51.403975 9499 log.cc:1196] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/6a31c07ade9844289468a67613b18ba8
I20260620 21:58:51.404232 9499 ts_tablet_manager.cc:1939] T 6a31c07ade9844289468a67613b18ba8 P 71a392e94dec40598081302d21f8f447: Deleting consensus metadata
I20260620 21:58:51.404575 8835 catalog_manager.cc:4841] TS 71a392e94dec40598081302d21f8f447 (127.6.241.131:45189): tablet 6a31c07ade9844289468a67613b18ba8 (table test-table [id=40281701be5c43889ebb32b945cad56d]) successfully deleted
I20260620 21:58:51.404698 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:51.402713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:41151 (local address 127.0.0.1:41252)
0620 21:58:51.404587 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:51.404650 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:41151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1749,"thread_start_us":75,"threads_started":1}
W20260620 21:58:51.404829 8836 catalog_manager.cc:4584] TS b049c8424a7346189c4e551d2f4abadd (127.6.241.129:41151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a31c07ade9844289468a67613b18ba8: Network error: Client connection negotiation failed: client connection to 127.6.241.129:41151: connect: Connection refused (error 111)
I20260620 21:58:51.408509 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 9201
I20260620 21:58:51.409263 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:51.408495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:36195 (local address 127.0.0.1:56408)
0620 21:58:51.408567 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:51.409209 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.130:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 21:58:51.414302 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 8817
2026-06-20T21:58:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6029 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:58:51 dist-test-slave-gl2g krb5kdc[9505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:58:51 dist-test-slave-gl2g krb5kdc[9505](info): set up 2 sockets
Jun 20 21:58:51 dist-test-slave-gl2g krb5kdc[9505](info): commencing operation
krb5kdc: starting...
W20260620 21:58:52.470430 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:58:52 dist-test-slave-gl2g krb5kdc[9505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:58:52.590494 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:44477
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:44477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:52.672201 9518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:52.672487 9518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:52.672528 9518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:52.673797 9518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:52.673882 9518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:52.673908 9518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:52.673928 9518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:52.673956 9518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:52.675616 9518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:44477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:52.675966 9518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:52.676215 9518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:52.681319 9523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:52.681319 9526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:52.681345 9524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:52.681740 9518 server_base.cc:1034] running on GCE node
I20260620 21:58:52.682036 9518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:52.682367 9518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:52.683544 9518 hybrid_clock.cc:648] HybridClock initialized: now 1781992732683521 us; error 32 us; skew 500 ppm
Jun 20 21:58:52 dist-test-slave-gl2g krb5kdc[9505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992732, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:52.686177 9518 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:58:52.686771 9518 webserver.cc:460] Webserver started at http://127.6.241.190:38409/ using document root <none> and password file <none>
I20260620 21:58:52.687075 9518 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:52.687151 9518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:52.687318 9518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:52.688453 9518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "a0dff2d8b2ca421abca560491ed01194"
format_stamp: "Formatted at 2026-06-20 21:58:52 on dist-test-slave-gl2g"
server_key: "d52f534c0f671d6ee8bf1019b16a52a7"
server_key_iv: "b1527c3552e09837bf816c009a793b89"
server_key_version: "encryptionkey@0"
I20260620 21:58:52.688853 9518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "a0dff2d8b2ca421abca560491ed01194"
format_stamp: "Formatted at 2026-06-20 21:58:52 on dist-test-slave-gl2g"
server_key: "d52f534c0f671d6ee8bf1019b16a52a7"
server_key_iv: "b1527c3552e09837bf816c009a793b89"
server_key_version: "encryptionkey@0"
I20260620 21:58:52.690310 9518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:58:52.691195 9533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:52.691376 9518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 21:58:52.691489 9518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "a0dff2d8b2ca421abca560491ed01194"
format_stamp: "Formatted at 2026-06-20 21:58:52 on dist-test-slave-gl2g"
server_key: "d52f534c0f671d6ee8bf1019b16a52a7"
server_key_iv: "b1527c3552e09837bf816c009a793b89"
server_key_version: "encryptionkey@0"
I20260620 21:58:52.691574 9518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:52.707530 9518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:52.710088 9518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:52.710224 9518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:52.714059 9518 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:44477
I20260620 21:58:52.714114 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:44477 every 8 connection(s)
I20260620 21:58:52.714490 9518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:52.715461 9586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:52.715778 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 9518
I20260620 21:58:52.715891 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:58:52.716094 7110 external_mini_cluster.cc:1428] Setting key ff057966254d3744c2953a339b40788d
I20260620 21:58:52.718708 9586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194: Bootstrap starting.
I20260620 21:58:52.719455 9586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:52.719750 9586 log.cc:824] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:52.720506 9586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194: No bootstrap required, opened a new log
I20260620 21:58:52.722872 9586 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dff2d8b2ca421abca560491ed01194" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 44477 } }
I20260620 21:58:52.723042 9586 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:52.723112 9586 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dff2d8b2ca421abca560491ed01194, State: Initialized, Role: FOLLOWER
Jun 20 21:58:52 dist-test-slave-gl2g krb5kdc[9505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:52.723290 9586 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dff2d8b2ca421abca560491ed01194" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 44477 } }
I20260620 21:58:52.723397 9586 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:52.723440 9586 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:52.723489 9586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:52.724195 9586 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dff2d8b2ca421abca560491ed01194" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 44477 } }
I20260620 21:58:52.724332 9586 leader_election.cc:302] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0dff2d8b2ca421abca560491ed01194; no voters:
I20260620 21:58:52.724507 9586 leader_election.cc:288] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:58:52.724596 9591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:52.724784 9586 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:52.724835 9591 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [term 1 LEADER]: Becoming Leader. State: Replica: a0dff2d8b2ca421abca560491ed01194, State: Running, Role: LEADER
I20260620 21:58:52.725024 9591 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dff2d8b2ca421abca560491ed01194" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 44477 } }
I20260620 21:58:52.725420 9592 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0dff2d8b2ca421abca560491ed01194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dff2d8b2ca421abca560491ed01194" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 44477 } } }
I20260620 21:58:52.725514 9592 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:52.725692 9593 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0dff2d8b2ca421abca560491ed01194. Latest consensus state: current_term: 1 leader_uuid: "a0dff2d8b2ca421abca560491ed01194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dff2d8b2ca421abca560491ed01194" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 44477 } } }
I20260620 21:58:52.725749 9593 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194 [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:52.725916 9600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:52.726475 9600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:52.728513 9600 catalog_manager.cc:1347] Generated new cluster ID: e6cf83d303674797a34e05904b92996b
I20260620 21:58:52.728529 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:52.716609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.6.241.190:44477)
0620 21:58:52.716874 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:52.716885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:52.716919 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:58:52.717455 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:52.717465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:52.717532 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:52.717689 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:52.718350 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:52.719744 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:52.720561 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:52.721157 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:52.721274 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:52.723887 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:52.723892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:52.723892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:52.723906 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:52.726715 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:52.727113 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:52.727117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:52.727117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:52.727251 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:52.727466 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:52.727467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:52.727467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:52.727734 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:52.727919 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:52.728184 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 21:58:52.728326 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":64,"threads_started":1}
I20260620 21:58:52.728576 9600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:58:52.739106 9600 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:58:52.739677 9600 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:52.745081 9600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0dff2d8b2ca421abca560491ed01194: Generated new TSK 0
I20260620 21:58:52.745297 9600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:58:52.778734 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:44477
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:52.881587 9612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:52.881806 9612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:52.881834 9612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:52.883281 9612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:52.883333 9612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:52.883364 9612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:58:52.884917 9612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:52.885213 9612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:52.885464 9612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:58:52.892108 9612 server_base.cc:1034] running on GCE node
W20260620 21:58:52.891994 9620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:52.891994 9617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:52.892063 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:52.892580 9612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:52.892902 9612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:52.894099 9612 hybrid_clock.cc:648] HybridClock initialized: now 1781992732894060 us; error 59 us; skew 500 ppm
Jun 20 21:58:52 dist-test-slave-gl2g krb5kdc[9505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992732, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:52.897574 9612 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:58:52.898123 9612 webserver.cc:460] Webserver started at http://127.6.241.129:45945/ using document root <none> and password file <none>
I20260620 21:58:52.898413 9612 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:52.898494 9612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:52.898697 9612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:52.899889 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "171dd1fa5221499cbd6ea028a097c062"
format_stamp: "Formatted at 2026-06-20 21:58:52 on dist-test-slave-gl2g"
server_key: "1e89bfa8d672cd4fa380a18a59bbfc30"
server_key_iv: "6ee534326407012bf4fd298e4d429610"
server_key_version: "encryptionkey@0"
I20260620 21:58:52.900297 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "171dd1fa5221499cbd6ea028a097c062"
format_stamp: "Formatted at 2026-06-20 21:58:52 on dist-test-slave-gl2g"
server_key: "1e89bfa8d672cd4fa380a18a59bbfc30"
server_key_iv: "6ee534326407012bf4fd298e4d429610"
server_key_version: "encryptionkey@0"
I20260620 21:58:52.901794 9612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:58:52.902689 9627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:52.902884 9612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:52.902989 9612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "171dd1fa5221499cbd6ea028a097c062"
format_stamp: "Formatted at 2026-06-20 21:58:52 on dist-test-slave-gl2g"
server_key: "1e89bfa8d672cd4fa380a18a59bbfc30"
server_key_iv: "6ee534326407012bf4fd298e4d429610"
server_key_version: "encryptionkey@0"
I20260620 21:58:52.903079 9612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:52.914561 9612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:52.916850 9612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:52.917040 9612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:52.917299 9612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:52.917644 9612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:52.917706 9612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:52.917743 9612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:52.917797 9612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:52.923678 9612 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:33651
I20260620 21:58:52.923755 9740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:33651 every 8 connection(s)
I20260620 21:58:52.924113 9612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:58:52.924750 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 9612
I20260620 21:58:52.924837 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:58:52.925051 7110 external_mini_cluster.cc:1428] Setting key 34a39582fc58e76589aa8ba07391d61a
Jun 20 21:58:52 dist-test-slave-gl2g krb5kdc[9505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992732, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
I20260620 21:58:52.934777 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:52.924954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:35813 (local address 127.6.241.190:44477)
0620 21:58:52.925012 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:52.925015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:52.925354 (+ 339us) server_negotiation.cc:407] Connection header received
0620 21:58:52.926071 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:52.926072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:52.926082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:52.926124 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:52.927541 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:52.928063 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:52.929119 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:52.929357 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:52.929445 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:52.931810 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:52.931811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:52.931811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:52.931820 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:52.933252 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:52.933702 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:52.933703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:52.933703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:52.933763 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:52.934034 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:52.934035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:52.934035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:52.934191 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:52.934265 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:52.934563 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 21:58:52.934676 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 21:58:52.934994 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:52.924958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:44477 (local address 127.6.241.129:35813)
0620 21:58:52.925200 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:52.925247 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:52.925970 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:52.926205 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:52.926209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:52.927104 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:52.927417 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:52.927426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:52.928165 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:52.928166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:52.929005 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:52.929008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:52.929456 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:52.929457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:52.929599 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:52.930210 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:52.930219 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:52.931696 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:52.933358 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:52.933367 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:52.933369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:52.933608 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:52.933840 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:52.933841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:52.933841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:52.933959 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:52.934271 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:52.934278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:52.934427 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 21:58:52.934542 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 21:58:52.934738 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
I20260620 21:58:52.935580 9741 heartbeater.cc:347] Connected to a master server at 127.6.241.190:44477
I20260620 21:58:52.935709 9741 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
I20260620 21:58:52.935952 9741 heartbeater.cc:511] Master 127.6.241.190:44477 requested a full tablet report, sending...
I20260620 21:58:52.936571 9550 ts_manager.cc:194] Registered new tserver with Master: 171dd1fa5221499cbd6ea028a097c062 (127.6.241.129:33651)
I20260620 21:58:52.937383 9550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:35813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:58:52.951759 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:44477
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:53.035580 9746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:53.035874 9746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:53.035916 9746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:53.037241 9746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:53.037317 9746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:53.037364 9746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:58:53.038986 9746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:53.039273 9746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:53.039528 9746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:53.044727 9751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:53.044781 9752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:53.044739 9754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:53.044739 9746 server_base.cc:1034] running on GCE node
I20260620 21:58:53.045117 9746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:53.045356 9746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:53.046494 9746 hybrid_clock.cc:648] HybridClock initialized: now 1781992733046482 us; error 33 us; skew 500 ppm
Jun 20 21:58:53 dist-test-slave-gl2g krb5kdc[9505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:53.049153 9746 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:58:53.049613 9746 webserver.cc:460] Webserver started at http://127.6.241.130:37799/ using document root <none> and password file <none>
I20260620 21:58:53.049880 9746 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:53.049929 9746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:53.050082 9746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:53.051257 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "fb555b24d75947159711da45e3b548e8"
format_stamp: "Formatted at 2026-06-20 21:58:53 on dist-test-slave-gl2g"
server_key: "e65492a85a8df27a5a4625fa1b6af5ab"
server_key_iv: "351e87b5f03c75af54c3c280b354e533"
server_key_version: "encryptionkey@0"
I20260620 21:58:53.051643 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "fb555b24d75947159711da45e3b548e8"
format_stamp: "Formatted at 2026-06-20 21:58:53 on dist-test-slave-gl2g"
server_key: "e65492a85a8df27a5a4625fa1b6af5ab"
server_key_iv: "351e87b5f03c75af54c3c280b354e533"
server_key_version: "encryptionkey@0"
I20260620 21:58:53.053076 9746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:58:53.053941 9761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:53.054217 9746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:53.054308 9746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "fb555b24d75947159711da45e3b548e8"
format_stamp: "Formatted at 2026-06-20 21:58:53 on dist-test-slave-gl2g"
server_key: "e65492a85a8df27a5a4625fa1b6af5ab"
server_key_iv: "351e87b5f03c75af54c3c280b354e533"
server_key_version: "encryptionkey@0"
I20260620 21:58:53.054404 9746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:53.075598 9746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:53.078050 9746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:53.078209 9746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:53.078474 9746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:53.078858 9746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:53.078917 9746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:53.078951 9746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:53.078984 9746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:53.084755 9746 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:33659
I20260620 21:58:53.084841 9874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:33659 every 8 connection(s)
I20260620 21:58:53.085186 9746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:58:53.086864 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 9746
I20260620 21:58:53.086979 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:58:53.087172 7110 external_mini_cluster.cc:1428] Setting key cc7eb88270a7d850706c0fd03140df81
Jun 20 21:58:53 dist-test-slave-gl2g krb5kdc[9505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:53.095920 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:53.085950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:44477 (local address 127.6.241.130:44171)
0620 21:58:53.086129 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:53.086177 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:53.086774 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:53.087030 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:53.087034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:53.087865 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:53.088126 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:53.088133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:53.088807 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:53.088808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:53.089718 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:53.089720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:53.090113 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:53.090114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:53.090230 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:53.090844 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:53.090852 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:53.092561 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:53.094462 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:53.094467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:53.094469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:53.094765 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:53.094970 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:53.094970 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:53.094970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:53.095069 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:53.095374 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:53.095379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:53.095512 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 21:58:53.095591 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:58:53.095732 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260620 21:58:53.095916 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:53.085992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:44171 (local address 127.6.241.190:44477)
0620 21:58:53.086082 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:53.086084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:53.086266 (+ 182us) server_negotiation.cc:407] Connection header received
0620 21:58:53.086882 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:53.086883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:53.086896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:53.086952 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:53.088221 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:53.088710 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:53.089817 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:53.090008 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:53.090097 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:53.092668 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:53.092670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:53.092671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:53.092680 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:53.094366 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:53.094841 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:53.094842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:53.094843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:53.094902 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:53.095144 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:53.095145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:53.095146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:53.095298 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:53.095369 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:53.095668 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 21:58:53.095769 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
I20260620 21:58:53.096602 9875 heartbeater.cc:347] Connected to a master server at 127.6.241.190:44477
I20260620 21:58:53.096740 9875 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:53.096930 9875 heartbeater.cc:511] Master 127.6.241.190:44477 requested a full tablet report, sending...
I20260620 21:58:53.097340 9550 ts_manager.cc:194] Registered new tserver with Master: fb555b24d75947159711da45e3b548e8 (127.6.241.130:33659)
I20260620 21:58:53.097739 9550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:44171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:58:53.113615 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:44477
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:53.195991 9880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:53.196202 9880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:53.196226 9880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:53.197418 9880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:53.197460 9880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:53.197515 9880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:58:53.199036 9880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:53.199282 9880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:53.199595 9880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:53.204991 9885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:53.204962 9886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:53.204991 9888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:53.205013 9880 server_base.cc:1034] running on GCE node
I20260620 21:58:53.205493 9880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:53.205809 9880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:53.206997 9880 hybrid_clock.cc:648] HybridClock initialized: now 1781992733206992 us; error 40 us; skew 500 ppm
Jun 20 21:58:53 dist-test-slave-gl2g krb5kdc[9505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:53.209808 9880 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:58:53.210318 9880 webserver.cc:460] Webserver started at http://127.6.241.131:44805/ using document root <none> and password file <none>
I20260620 21:58:53.210575 9880 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:53.210711 9880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:53.210872 9880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:53.212033 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "5d731211b1cf4ea59b0dce77395f97d6"
format_stamp: "Formatted at 2026-06-20 21:58:53 on dist-test-slave-gl2g"
server_key: "45a4ba14dc5fa514fc8342c964bcde27"
server_key_iv: "1bd85c3ac2ce1a10c0d206d3c19aff16"
server_key_version: "encryptionkey@0"
I20260620 21:58:53.212430 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "5d731211b1cf4ea59b0dce77395f97d6"
format_stamp: "Formatted at 2026-06-20 21:58:53 on dist-test-slave-gl2g"
server_key: "45a4ba14dc5fa514fc8342c964bcde27"
server_key_iv: "1bd85c3ac2ce1a10c0d206d3c19aff16"
server_key_version: "encryptionkey@0"
I20260620 21:58:53.213837 9880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:58:53.214708 9895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:53.214890 9880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:53.214984 9880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "5d731211b1cf4ea59b0dce77395f97d6"
format_stamp: "Formatted at 2026-06-20 21:58:53 on dist-test-slave-gl2g"
server_key: "45a4ba14dc5fa514fc8342c964bcde27"
server_key_iv: "1bd85c3ac2ce1a10c0d206d3c19aff16"
server_key_version: "encryptionkey@0"
I20260620 21:58:53.215080 9880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:53.228480 9880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:53.231020 9880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:53.231186 9880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:53.231436 9880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:53.231774 9880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:53.231837 9880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:53.231878 9880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:53.231910 9880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:53.237958 9880 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:34711
I20260620 21:58:53.237998 10008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:34711 every 8 connection(s)
I20260620 21:58:53.238346 9880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
Jun 20 21:58:53 dist-test-slave-gl2g krb5kdc[9505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:53.248442 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:53.239144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:49401 (local address 127.6.241.190:44477)
0620 21:58:53.239212 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:53.239216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:53.239433 (+ 217us) server_negotiation.cc:407] Connection header received
0620 21:58:53.239963 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:53.239964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:53.239975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:53.240034 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:53.241038 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:53.241894 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:53.243206 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:53.243411 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:53.243476 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:53.245839 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:53.245841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:53.245841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:53.245848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:53.247168 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:53.247551 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:53.247552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:53.247552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:53.247603 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:53.247821 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:53.247822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:53.247822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:53.247929 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:53.247981 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:53.248219 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 21:58:53.248280 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 21:58:53.248500 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 9880
I20260620 21:58:53.248512 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:53.239132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:44477 (local address 127.6.241.131:49401)
0620 21:58:53.239294 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:53.239342 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:53.239842 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:53.240093 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:53.240097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:53.240689 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:53.240916 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:53.240921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:53.242034 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:53.242035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:53.243122 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:53.243124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:53.243521 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:53.243522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:53.243637 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:53.244262 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:53.244272 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:53.245749 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:53.247245 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:53.247251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:53.247254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:53.247483 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:53.247662 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:53.247662 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:53.247662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:53.247764 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:53.247983 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:53.247987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:53.248107 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 21:58:53.248190 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 21:58:53.248313 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260620 21:58:53.248574 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:58:53.248771 7110 external_mini_cluster.cc:1428] Setting key 6f8e903ef6758f3ed6a968e34e96f40d
I20260620 21:58:53.249070 10009 heartbeater.cc:347] Connected to a master server at 127.6.241.190:44477
I20260620 21:58:53.249182 10009 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:53.249398 10009 heartbeater.cc:511] Master 127.6.241.190:44477 requested a full tablet report, sending...
I20260620 21:58:53.249758 9550 ts_manager.cc:194] Registered new tserver with Master: 5d731211b1cf4ea59b0dce77395f97d6 (127.6.241.131:34711)
I20260620 21:58:53.250135 9550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:49401
I20260620 21:58:53.260766 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:58:53.271792 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:58:53.270817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.6.241.190:44477)
0620 21:58:53.270892 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:53.270895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:53.271047 (+ 152us) server_negotiation.cc:407] Connection header received
0620 21:58:53.271093 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:53.271094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:53.271104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:53.271156 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:53.271716 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56758: BlockingRecv error: recv got EOF from 127.0.0.1:56758 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:58:53.272605 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 9612
I20260620 21:58:53.278751 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 9746
I20260620 21:58:53.283953 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 9880
I20260620 21:58:53.288839 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 9518
2026-06-20T21:58:53Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1873 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:58:53 dist-test-slave-gl2g krb5kdc[10021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:58:53 dist-test-slave-gl2g krb5kdc[10021](info): set up 2 sockets
Jun 20 21:58:53 dist-test-slave-gl2g krb5kdc[10021](info): commencing operation
krb5kdc: starting...
W20260620 21:58:54.334506 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:58:54 dist-test-slave-gl2g krb5kdc[10021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:58:54.453441 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41737
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:41737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:54.536079 10034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:54.536314 10034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:54.536340 10034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:54.537573 10034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:54.537623 10034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:54.537638 10034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:54.537652 10034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:54.537716 10034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:54.539322 10034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:41737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:54.539599 10034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:54.539858 10034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:54.545015 10040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:54.545116 10034 server_base.cc:1034] running on GCE node
W20260620 21:58:54.545049 10042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:54.545032 10039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:54.545490 10034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:54.545807 10034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:54.546975 10034 hybrid_clock.cc:648] HybridClock initialized: now 1781992734546964 us; error 31 us; skew 500 ppm
Jun 20 21:58:54 dist-test-slave-gl2g krb5kdc[10021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992734, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:54.554731 10034 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:58:54.555197 10034 webserver.cc:460] Webserver started at http://127.6.241.190:35897/ using document root <none> and password file <none>
I20260620 21:58:54.555414 10034 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:54.555460 10034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:54.555557 10034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:54.556721 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "72c4f2a64c4744c9809ae8278cdd2479"
format_stamp: "Formatted at 2026-06-20 21:58:54 on dist-test-slave-gl2g"
server_key: "dd0395bfe67a82afe5efcb6b2f8842c4"
server_key_iv: "0136dafd103d72bd3f8587cd8750dfd9"
server_key_version: "encryptionkey@0"
I20260620 21:58:54.557063 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "72c4f2a64c4744c9809ae8278cdd2479"
format_stamp: "Formatted at 2026-06-20 21:58:54 on dist-test-slave-gl2g"
server_key: "dd0395bfe67a82afe5efcb6b2f8842c4"
server_key_iv: "0136dafd103d72bd3f8587cd8750dfd9"
server_key_version: "encryptionkey@0"
I20260620 21:58:54.558507 10034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:58:54.559408 10049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:54.559589 10034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:54.559672 10034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "72c4f2a64c4744c9809ae8278cdd2479"
format_stamp: "Formatted at 2026-06-20 21:58:54 on dist-test-slave-gl2g"
server_key: "dd0395bfe67a82afe5efcb6b2f8842c4"
server_key_iv: "0136dafd103d72bd3f8587cd8750dfd9"
server_key_version: "encryptionkey@0"
I20260620 21:58:54.559736 10034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:54.570864 10034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:54.572801 10034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:54.572932 10034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:54.577521 10034 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:41737
I20260620 21:58:54.577589 10101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:41737 every 8 connection(s)
I20260620 21:58:54.577967 10034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:54.578698 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 10034
I20260620 21:58:54.578800 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:58:54.579017 7110 external_mini_cluster.cc:1428] Setting key f729bf95cc50a885cfc5e14105a268ee
I20260620 21:58:54.579077 10102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:54.582669 10102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479: Bootstrap starting.
I20260620 21:58:54.583410 10102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:54.583715 10102 log.cc:824] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:54.584450 10102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479: No bootstrap required, opened a new log
Jun 20 21:58:54 dist-test-slave-gl2g krb5kdc[10021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:54.586730 10102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c4f2a64c4744c9809ae8278cdd2479" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41737 } }
I20260620 21:58:54.586907 10102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:54.586973 10102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c4f2a64c4744c9809ae8278cdd2479, State: Initialized, Role: FOLLOWER
I20260620 21:58:54.587148 10102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c4f2a64c4744c9809ae8278cdd2479" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41737 } }
I20260620 21:58:54.587244 10102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:54.587288 10102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:54.587339 10102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:54.587898 10102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c4f2a64c4744c9809ae8278cdd2479" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41737 } }
I20260620 21:58:54.588016 10102 leader_election.cc:302] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72c4f2a64c4744c9809ae8278cdd2479; no voters:
I20260620 21:58:54.588227 10102 leader_election.cc:288] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:58:54.588284 10107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:54.588467 10107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [term 1 LEADER]: Becoming Leader. State: Replica: 72c4f2a64c4744c9809ae8278cdd2479, State: Running, Role: LEADER
I20260620 21:58:54.588527 10102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:54.588615 10107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c4f2a64c4744c9809ae8278cdd2479" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41737 } }
I20260620 21:58:54.589218 10107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72c4f2a64c4744c9809ae8278cdd2479. Latest consensus state: current_term: 1 leader_uuid: "72c4f2a64c4744c9809ae8278cdd2479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c4f2a64c4744c9809ae8278cdd2479" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41737 } } }
I20260620 21:58:54.589308 10107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:54.589223 10109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72c4f2a64c4744c9809ae8278cdd2479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c4f2a64c4744c9809ae8278cdd2479" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41737 } } }
I20260620 21:58:54.589599 10109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479 [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:54.589962 10116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:54.590667 10116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:54.590834 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:54.579559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.6.241.190:41737)
0620 21:58:54.579749 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:54.579758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:54.579784 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:58:54.580233 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:54.580240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:54.580288 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:54.580418 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:54.581249 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:54.582148 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:54.583243 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:54.583717 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:54.583881 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:54.586572 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:54.586576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:54.586577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:54.586588 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:54.588837 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:54.589274 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:54.589277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:54.589278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:54.589407 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:54.589626 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:54.589627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:54.589628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:54.589959 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:54.590121 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:54.590406 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 21:58:54.590551 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260620 21:58:54.591730 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:58:54.602921 10116 catalog_manager.cc:1347] Generated new cluster ID: baae477bf3da4586829561d4122d713b
I20260620 21:58:54.603012 10116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:58:54.607510 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:54.602641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.6.241.190:41737)
0620 21:58:54.602693 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:54.602695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:54.603044 (+ 349us) server_negotiation.cc:407] Connection header received
0620 21:58:54.603162 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:54.603163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:54.603179 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:54.603250 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:54.603698 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:54.604468 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:54.605560 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:54.605891 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:54.605967 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:54.606986 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:54.606987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:54.606988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:58:54.607222 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:54.607242 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:58:54.607248 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:54.607308 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:54.607395 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 21:58:54.607424 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:58:54.623216 10116 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:58:54.623790 10116 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:54.627118 10116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72c4f2a64c4744c9809ae8278cdd2479: Generated new TSK 0
I20260620 21:58:54.627388 10116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:58:54.630681 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 10034
2026-06-20T21:58:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1339 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:58:54 dist-test-slave-gl2g krb5kdc[10135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:58:54 dist-test-slave-gl2g krb5kdc[10135](info): set up 2 sockets
Jun 20 21:58:54 dist-test-slave-gl2g krb5kdc[10135](info): commencing operation
krb5kdc: starting...
W20260620 21:58:55.677670 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 21:58:55.921435 7110 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.244s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 21:58:56.141637 7110 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.220s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 21:58:56.446451 7110 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.305s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/test-user.keytab.
W20260620 21:58:57.481223 7110 mini_kdc.cc:302] Time spent creating keytab for test-user: real 1.035s user 0.000s sys 0.002s
Jun 20 21:58:57 dist-test-slave-gl2g krb5kdc[10135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
W20260620 21:58:58.859019 7110 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.241.190: real 1.354s user 0.000s sys 0.003s
I20260620 21:58:58.859452 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:35275
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:58.942320 10148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:58.942608 10148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:58.942675 10148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:58.943903 10148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:58.943984 10148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:58.944012 10148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:58.944046 10148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:58.944067 10148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:58.945606 10148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:35275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:58.945899 10148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:58.946139 10148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:58.951217 10156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:58.951409 10153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:58.951520 10154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:58.951596 10148 server_base.cc:1034] running on GCE node
I20260620 21:58:58.951871 10148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:58.952224 10148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:58.953424 10148 hybrid_clock.cc:648] HybridClock initialized: now 1781992738953404 us; error 49 us; skew 500 ppm
Jun 20 21:58:58 dist-test-slave-gl2g krb5kdc[10135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992738, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:58.956197 10148 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:58:58.956720 10148 webserver.cc:460] Webserver started at http://127.6.241.190:41345/ using document root <none> and password file <none>
I20260620 21:58:58.956985 10148 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:58.957059 10148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:58.957211 10148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:58.958531 10148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "e22f5adda5ea47c2a77f128f4769abcd"
format_stamp: "Formatted at 2026-06-20 21:58:58 on dist-test-slave-gl2g"
server_key: "a289ebae0dc9fdc8032ce35503226076"
server_key_iv: "14954344f62180c7b1d7f805ed22e802"
server_key_version: "encryptionkey@0"
I20260620 21:58:58.959028 10148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "e22f5adda5ea47c2a77f128f4769abcd"
format_stamp: "Formatted at 2026-06-20 21:58:58 on dist-test-slave-gl2g"
server_key: "a289ebae0dc9fdc8032ce35503226076"
server_key_iv: "14954344f62180c7b1d7f805ed22e802"
server_key_version: "encryptionkey@0"
I20260620 21:58:58.960932 10148 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:58:58.961853 10163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:58.962024 10148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:58.962141 10148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "e22f5adda5ea47c2a77f128f4769abcd"
format_stamp: "Formatted at 2026-06-20 21:58:58 on dist-test-slave-gl2g"
server_key: "a289ebae0dc9fdc8032ce35503226076"
server_key_iv: "14954344f62180c7b1d7f805ed22e802"
server_key_version: "encryptionkey@0"
I20260620 21:58:58.962244 10148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:58.975335 10148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:58.977372 10148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:58.977547 10148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:58.981726 10148 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:35275
I20260620 21:58:58.981762 10215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:35275 every 8 connection(s)
I20260620 21:58:58.982133 10148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:58.983228 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:58.984385 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 10148
I20260620 21:58:58.984491 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:58:58.984674 7110 external_mini_cluster.cc:1428] Setting key 88a3c18427e3d7e22906c97f29084a5c
I20260620 21:58:58.986192 10216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd: Bootstrap starting.
I20260620 21:58:58.987028 10216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:58.987336 10216 log.cc:824] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:58.988090 10216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd: No bootstrap required, opened a new log
I20260620 21:58:58.990047 10216 raft_consensus.cc:348] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f5adda5ea47c2a77f128f4769abcd" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35275 } }
I20260620 21:58:58.990239 10216 raft_consensus.cc:374] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:58.990289 10216 raft_consensus.cc:729] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22f5adda5ea47c2a77f128f4769abcd, State: Initialized, Role: FOLLOWER
I20260620 21:58:58.990437 10216 consensus_queue.cc:260] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f5adda5ea47c2a77f128f4769abcd" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35275 } }
I20260620 21:58:58.990533 10216 raft_consensus.cc:388] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:58.990577 10216 raft_consensus.cc:482] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:58.990653 10216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [term 0 FOLLOWER]: Advancing to term 1
Jun 20 21:58:58 dist-test-slave-gl2g krb5kdc[10135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:58.995890 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:58.985240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.6.241.190:35275)
0620 21:58:58.985421 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:58.985428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:58.985454 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:58:58.985925 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:58.985932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:58.985980 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:58.986104 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:58.986936 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:58.988154 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:58.989024 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:58.989465 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:58.989590 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:58.992717 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:58.992723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:58.992728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:58.992740 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:58.994260 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:58.994703 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:58.994706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:58.994707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:58.994801 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:58.995019 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:58.995019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:58.995020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:58.995272 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:58.995461 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:58.995547 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 21:58:58.995697 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260620 21:58:59.029395 10216 raft_consensus.cc:504] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f5adda5ea47c2a77f128f4769abcd" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35275 } }
I20260620 21:58:59.029608 10216 leader_election.cc:302] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e22f5adda5ea47c2a77f128f4769abcd; no voters:
I20260620 21:58:59.029847 10216 leader_election.cc:288] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:58:59.030006 10221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:59.030141 10216 sys_catalog.cc:564] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:59.030234 10221 raft_consensus.cc:686] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [term 1 LEADER]: Becoming Leader. State: Replica: e22f5adda5ea47c2a77f128f4769abcd, State: Running, Role: LEADER
I20260620 21:58:59.030373 10221 consensus_queue.cc:237] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f5adda5ea47c2a77f128f4769abcd" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35275 } }
I20260620 21:58:59.030879 10222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e22f5adda5ea47c2a77f128f4769abcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f5adda5ea47c2a77f128f4769abcd" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35275 } } }
I20260620 21:58:59.030990 10222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:59.031167 10223 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e22f5adda5ea47c2a77f128f4769abcd. Latest consensus state: current_term: 1 leader_uuid: "e22f5adda5ea47c2a77f128f4769abcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f5adda5ea47c2a77f128f4769abcd" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35275 } } }
I20260620 21:58:59.031224 10223 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:59.031399 10230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:59.031983 10230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:59.033648 10230 catalog_manager.cc:1347] Generated new cluster ID: 5a5fd89619534f81928e3f9bd2679057
I20260620 21:58:59.033694 10230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:58:59.038439 10230 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:58:59.039041 10230 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:59.045130 10230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e22f5adda5ea47c2a77f128f4769abcd: Generated new TSK 0
I20260620 21:58:59.045367 10230 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
W20260620 21:58:59.425326 7110 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.241.129: real 0.366s user 0.000s sys 0.003s
I20260620 21:58:59.425662 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:35275
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:59.508889 10242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:59.509130 10242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:59.509161 10242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:59.510406 10242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:59.510454 10242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:59.510519 10242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:58:59.512069 10242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:59.512331 10242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:59.512593 10242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:59.517803 10247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:59.517867 10242 server_base.cc:1034] running on GCE node
W20260620 21:58:59.517814 10248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:59.517814 10250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:59.518224 10242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:59.518503 10242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:59.519683 10242 hybrid_clock.cc:648] HybridClock initialized: now 1781992739519697 us; error 58 us; skew 500 ppm
Jun 20 21:58:59 dist-test-slave-gl2g krb5kdc[10135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:59.522312 10242 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:58:59.522871 10242 webserver.cc:460] Webserver started at http://127.6.241.129:34403/ using document root <none> and password file <none>
I20260620 21:58:59.523125 10242 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:59.523195 10242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:59.523355 10242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:59.524463 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "91f892149fea4732a3eba621607cdc67"
format_stamp: "Formatted at 2026-06-20 21:58:59 on dist-test-slave-gl2g"
server_key: "0969b959148d518e39d6e84bfdfd526d"
server_key_iv: "1d6a911df939d96fe18cbbe5740cea4c"
server_key_version: "encryptionkey@0"
I20260620 21:58:59.524835 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "91f892149fea4732a3eba621607cdc67"
format_stamp: "Formatted at 2026-06-20 21:58:59 on dist-test-slave-gl2g"
server_key: "0969b959148d518e39d6e84bfdfd526d"
server_key_iv: "1d6a911df939d96fe18cbbe5740cea4c"
server_key_version: "encryptionkey@0"
I20260620 21:58:59.526216 10242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:58:59.527031 10257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.527209 10242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.527318 10242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "91f892149fea4732a3eba621607cdc67"
format_stamp: "Formatted at 2026-06-20 21:58:59 on dist-test-slave-gl2g"
server_key: "0969b959148d518e39d6e84bfdfd526d"
server_key_iv: "1d6a911df939d96fe18cbbe5740cea4c"
server_key_version: "encryptionkey@0"
I20260620 21:58:59.527397 10242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:59.564493 10242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:59.567116 10242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:59.567293 10242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:59.567589 10242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:59.567966 10242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:59.568024 10242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.568066 10242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:59.568095 10242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.574209 10242 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:41497
I20260620 21:58:59.574251 10370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:41497 every 8 connection(s)
I20260620 21:58:59.574625 10242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:58:59 dist-test-slave-gl2g krb5kdc[10135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:59.581099 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 10242
I20260620 21:58:59.581200 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:58:59.581426 7110 external_mini_cluster.cc:1428] Setting key 234393733ea77ba413fcc261d7d77847
I20260620 21:58:59.585125 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:59.575410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:53947 (local address 127.6.241.190:35275)
0620 21:58:59.575564 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:59.575566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:59.575730 (+ 164us) server_negotiation.cc:407] Connection header received
0620 21:58:59.576320 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:59.576321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:59.576335 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:59.576399 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:59.577420 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.577949 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.579067 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.579270 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.579356 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:59.581803 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:59.581804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:59.581804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:59.581811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:59.583483 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:59.584018 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:59.584018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:59.584019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:59.584073 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:59.584371 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:59.584371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:59.584372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:59.584528 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:59.584597 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:59.584871 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 21:58:59.584971 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260620 21:58:59.585170 10374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:59.575390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:35275 (local address 127.6.241.129:53947)
0620 21:58:59.575575 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:59.575626 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:59.576199 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:59.576487 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:59.576496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:59.577095 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:59.577308 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:59.577313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.578055 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.578056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:59.578968 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:59.578971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.579370 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.579371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:59.579502 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:59.580053 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:59.580062 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:59.581682 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:59.583571 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:59.583579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:59.583582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:59.583930 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:59.584142 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:59.584143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:59.584144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:59.584301 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:59.584601 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:59.584607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:59.584734 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 21:58:59.584830 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:58:59.584989 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260620 21:58:59.585706 10371 heartbeater.cc:347] Connected to a master server at 127.6.241.190:35275
I20260620 21:58:59.585822 10371 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:59.586043 10371 heartbeater.cc:511] Master 127.6.241.190:35275 requested a full tablet report, sending...
I20260620 21:58:59.586690 10180 ts_manager.cc:194] Registered new tserver with Master: 91f892149fea4732a3eba621607cdc67 (127.6.241.129:41497)
I20260620 21:58:59.587693 10180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:53947
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:58:59.609007 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:35275
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:59.693964 10377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:59.694187 10377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:59.694216 10377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:59.695540 10377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:59.695591 10377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:59.695621 10377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:58:59.697245 10377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:59.697522 10377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:59.697781 10377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:59.703446 10382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:59.703500 10377 server_base.cc:1034] running on GCE node
W20260620 21:58:59.703503 10383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:59.703439 10385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:59.703804 10377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:59.704079 10377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:59.705225 10377 hybrid_clock.cc:648] HybridClock initialized: now 1781992739705206 us; error 46 us; skew 500 ppm
Jun 20 21:58:59 dist-test-slave-gl2g krb5kdc[10135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:59.707890 10377 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:58:59.708416 10377 webserver.cc:460] Webserver started at http://127.6.241.130:36659/ using document root <none> and password file <none>
I20260620 21:58:59.708658 10377 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:59.708729 10377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:59.708884 10377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:59.709990 10377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "25cffb637f30479aab967142b2b656ed"
format_stamp: "Formatted at 2026-06-20 21:58:59 on dist-test-slave-gl2g"
server_key: "3b07079f5ced9c2cba1660829ec6e32c"
server_key_iv: "79791f6298d713c13f30fdc14b847543"
server_key_version: "encryptionkey@0"
I20260620 21:58:59.710368 10377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "25cffb637f30479aab967142b2b656ed"
format_stamp: "Formatted at 2026-06-20 21:58:59 on dist-test-slave-gl2g"
server_key: "3b07079f5ced9c2cba1660829ec6e32c"
server_key_iv: "79791f6298d713c13f30fdc14b847543"
server_key_version: "encryptionkey@0"
I20260620 21:58:59.711869 10377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:58:59.712656 10392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.712831 10377 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:59.712950 10377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "25cffb637f30479aab967142b2b656ed"
format_stamp: "Formatted at 2026-06-20 21:58:59 on dist-test-slave-gl2g"
server_key: "3b07079f5ced9c2cba1660829ec6e32c"
server_key_iv: "79791f6298d713c13f30fdc14b847543"
server_key_version: "encryptionkey@0"
I20260620 21:58:59.713039 10377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:59.724362 10377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:59.726537 10377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:59.726738 10377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:59.726997 10377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:59.727324 10377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:59.727378 10377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.727423 10377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:59.727444 10377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.733924 10377 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:38401
I20260620 21:58:59.734046 10505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:38401 every 8 connection(s)
I20260620 21:58:59.734390 10377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
Jun 20 21:58:59 dist-test-slave-gl2g krb5kdc[10135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:59.743616 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:59.735213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:41435 (local address 127.6.241.190:35275)
0620 21:58:59.735308 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:59.735310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:59.735501 (+ 191us) server_negotiation.cc:407] Connection header received
0620 21:58:59.736023 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:59.736024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:59.736034 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:59.736087 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:59.737057 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.737556 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.738548 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.738760 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.738810 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:59.740985 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:59.740987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:59.740987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:59.740994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:59.742433 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:59.742846 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:59.742847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:59.742847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:59.742897 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:59.743100 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:59.743101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:59.743101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:59.743198 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:59.743244 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:59.743475 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 21:58:59.743545 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:58:59.743798 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:59.735172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:35275 (local address 127.6.241.130:41435)
0620 21:58:59.735357 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:59.735397 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:59.735926 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:59.736154 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:59.736157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:59.736769 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:59.736976 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:59.736981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.737638 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.737638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:59.738472 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:59.738474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.738829 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.738829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:59.738948 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:59.739484 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:59.739492 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:59.740878 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:59.742504 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:59.742509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:59.742511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:59.742786 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:59.742960 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:59.742961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:59.742961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:59.743060 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:59.743248 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:59.743252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:59.743368 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 21:58:59.743462 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 21:58:59.743582 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260620 21:58:59.744331 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 10377
I20260620 21:58:59.744344 10506 heartbeater.cc:347] Connected to a master server at 127.6.241.190:35275
I20260620 21:58:59.744436 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:58:59.744453 10506 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:59.744647 7110 external_mini_cluster.cc:1428] Setting key 112d2db576c7b606903c4aa8b4ecc906
I20260620 21:58:59.744666 10506 heartbeater.cc:511] Master 127.6.241.190:35275 requested a full tablet report, sending...
I20260620 21:58:59.745083 10180 ts_manager.cc:194] Registered new tserver with Master: 25cffb637f30479aab967142b2b656ed (127.6.241.130:38401)
I20260620 21:58:59.745462 10180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:41435
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:58:59.770287 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:35275
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:59.852371 10511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:59.852596 10511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:59.852620 10511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:59.853830 10511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:59.853875 10511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:59.853904 10511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:58:59.855409 10511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:58:59.855687 10511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:59.855953 10511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:59.861176 10516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:59.861178 10519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:59.861255 10511 server_base.cc:1034] running on GCE node
W20260620 21:58:59.861210 10517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:59.861605 10511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:59.861878 10511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:59.863019 10511 hybrid_clock.cc:648] HybridClock initialized: now 1781992739863006 us; error 26 us; skew 500 ppm
Jun 20 21:58:59 dist-test-slave-gl2g krb5kdc[10135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:59.865561 10511 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:58:59.866040 10511 webserver.cc:460] Webserver started at http://127.6.241.131:34569/ using document root <none> and password file <none>
I20260620 21:58:59.866246 10511 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:59.866287 10511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:59.866406 10511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:59.867544 10511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "27849727c819415a8f87cb4caf11dbbc"
format_stamp: "Formatted at 2026-06-20 21:58:59 on dist-test-slave-gl2g"
server_key: "54415fb3b415ebe017247c9165eb553a"
server_key_iv: "8387391e3a64e8bfb1b79b0d46f2f09c"
server_key_version: "encryptionkey@0"
I20260620 21:58:59.867859 10511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "27849727c819415a8f87cb4caf11dbbc"
format_stamp: "Formatted at 2026-06-20 21:58:59 on dist-test-slave-gl2g"
server_key: "54415fb3b415ebe017247c9165eb553a"
server_key_iv: "8387391e3a64e8bfb1b79b0d46f2f09c"
server_key_version: "encryptionkey@0"
I20260620 21:58:59.869143 10511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:58:59.869858 10526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.870045 10511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:58:59.870152 10511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "27849727c819415a8f87cb4caf11dbbc"
format_stamp: "Formatted at 2026-06-20 21:58:59 on dist-test-slave-gl2g"
server_key: "54415fb3b415ebe017247c9165eb553a"
server_key_iv: "8387391e3a64e8bfb1b79b0d46f2f09c"
server_key_version: "encryptionkey@0"
I20260620 21:58:59.870234 10511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:59.878369 10511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:59.880357 10511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:59.880537 10511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:59.880788 10511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:59.881114 10511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:59.881167 10511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.881209 10511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:59.881237 10511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:59.887317 10511 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:33177
I20260620 21:58:59.887393 10639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:33177 every 8 connection(s)
I20260620 21:58:59.887739 10511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
Jun 20 21:58:59 dist-test-slave-gl2g krb5kdc[10135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:59.895351 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 10511
I20260620 21:58:59.895442 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:58:59.895643 7110 external_mini_cluster.cc:1428] Setting key 7e6b75999e3fc1ca3d0e56bb4fc17f10
I20260620 21:58:59.897398 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:59.888472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:34787 (local address 127.6.241.190:35275)
0620 21:58:59.888554 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:59.888556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:59.888717 (+ 161us) server_negotiation.cc:407] Connection header received
0620 21:58:59.889196 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:59.889197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:59.889208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:59.889258 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:59.890218 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.890809 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.891809 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.892002 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.892053 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:59.894186 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:59.894187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:59.894188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:59.894195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:59.896021 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:59.896501 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:59.896502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:59.896502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:59.896590 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:59.896874 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:59.896875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:59.896875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:59.896990 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:59.897050 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:59.897256 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 21:58:59.897319 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:58:59.897642 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:59.888429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:35275 (local address 127.6.241.131:34787)
0620 21:58:59.888592 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:59.888630 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:59.889106 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:59.889323 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:59.889326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:59.889923 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:59.890130 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:59.890135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.890887 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.890888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:59.891729 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:59.891731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.892073 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.892073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:59.892173 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:59.892679 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:59.892687 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:59.894074 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:59.896082 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:59.896091 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:59.896094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:59.896426 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:59.896679 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:59.896679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:59.896680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:59.896810 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:59.897044 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:59.897046 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:59.897166 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 21:58:59.897245 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:58:59.897371 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260620 21:58:59.898129 10640 heartbeater.cc:347] Connected to a master server at 127.6.241.190:35275
I20260620 21:58:59.898214 10640 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:59.898361 10640 heartbeater.cc:511] Master 127.6.241.190:35275 requested a full tablet report, sending...
I20260620 21:58:59.898772 10180 ts_manager.cc:194] Registered new tserver with Master: 27849727c819415a8f87cb4caf11dbbc (127.6.241.131:33177)
I20260620 21:58:59.899161 10180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:34787
I20260620 21:58:59.907537 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:58:59 dist-test-slave-gl2g krb5kdc[10135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992739, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 21:58:59 dist-test-slave-gl2g krb5kdc[10135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992739, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:58:59.930737 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:59.923416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.6.241.190:35275)
0620 21:58:59.923491 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:59.923493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:59.923598 (+ 105us) server_negotiation.cc:407] Connection header received
0620 21:58:59.923693 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:59.923694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:59.923705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:59.923764 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:59.924331 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.924838 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.925699 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:59.925890 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:59.925980 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:59.928235 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:59.928237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:59.928237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:59.928245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:59.929425 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:59.929784 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:59.929785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:59.929786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:59.929844 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:59.930085 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:59.930086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:59.930086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:59.930304 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:59.930418 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:59.930522 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 21:58:59.930629 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 21:58:59.931149 10180 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52918
I20260620 21:58:59.931618 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 10242
I20260620 21:58:59.935979 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 10377
I20260620 21:58:59.940409 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 10511
I20260620 21:58:59.944761 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 10148
2026-06-20T21:58:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5316 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:00 dist-test-slave-gl2g krb5kdc[10652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:00 dist-test-slave-gl2g krb5kdc[10652](info): set up 2 sockets
Jun 20 21:59:00 dist-test-slave-gl2g krb5kdc[10652](info): commencing operation
krb5kdc: starting...
W20260620 21:59:00.990720 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:59:01.114141 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33545
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:33545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:01.197284 10665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:01.197535 10665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:01.197561 10665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:01.198902 10665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:01.198944 10665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:01.198961 10665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:01.198973 10665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:01.198985 10665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:01.200573 10665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:33545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:01.200837 10665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:01.201097 10665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:01.206342 10673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.206338 10671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.206342 10670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:01.206363 10665 server_base.cc:1034] running on GCE node
I20260620 21:59:01.206862 10665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:01.207192 10665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:01.208343 10665 hybrid_clock.cc:648] HybridClock initialized: now 1781992741208328 us; error 32 us; skew 500 ppm
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:01.210973 10665 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:59:01.211385 10665 webserver.cc:460] Webserver started at http://127.6.241.190:45355/ using document root <none> and password file <none>
I20260620 21:59:01.211573 10665 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:01.211616 10665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:01.211709 10665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:01.212776 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "d1cb67196292491b8760ec4c2c04e99b"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "c7aba053451e8b7c9bd86b89736fa0aa"
server_key_iv: "cda51d4afec7a1b5570cfb6d77bdc927"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.213096 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "d1cb67196292491b8760ec4c2c04e99b"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "c7aba053451e8b7c9bd86b89736fa0aa"
server_key_iv: "cda51d4afec7a1b5570cfb6d77bdc927"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.214391 10665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:01.215157 10680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.215327 10665 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.215438 10665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "d1cb67196292491b8760ec4c2c04e99b"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "c7aba053451e8b7c9bd86b89736fa0aa"
server_key_iv: "cda51d4afec7a1b5570cfb6d77bdc927"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.215517 10665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:01.233111 10665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:01.235364 10665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:01.235556 10665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:01.239651 10665 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:33545
I20260620 21:59:01.239689 10732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:33545 every 8 connection(s)
I20260620 21:59:01.240226 10665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:01.240945 10733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:01.243427 10733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b: Bootstrap starting.
I20260620 21:59:01.244024 10733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:01.244277 10733 log.cc:824] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:01.244998 10733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b: No bootstrap required, opened a new log
I20260620 21:59:01.246680 10733 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cb67196292491b8760ec4c2c04e99b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33545 } }
I20260620 21:59:01.246862 10733 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:01.246887 10733 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1cb67196292491b8760ec4c2c04e99b, State: Initialized, Role: FOLLOWER
I20260620 21:59:01.247045 10733 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cb67196292491b8760ec4c2c04e99b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33545 } }
I20260620 21:59:01.247133 10733 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:01.247157 10733 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:01.247187 10733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:01.248122 10733 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cb67196292491b8760ec4c2c04e99b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33545 } }
I20260620 21:59:01.248217 10733 leader_election.cc:302] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1cb67196292491b8760ec4c2c04e99b; no voters:
I20260620 21:59:01.248405 10733 leader_election.cc:288] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:01.248432 10736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:01.248631 10736 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [term 1 LEADER]: Becoming Leader. State: Replica: d1cb67196292491b8760ec4c2c04e99b, State: Running, Role: LEADER
I20260620 21:59:01.248692 10733 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:01.248850 10736 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cb67196292491b8760ec4c2c04e99b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33545 } }
I20260620 21:59:01.249270 10738 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1cb67196292491b8760ec4c2c04e99b. Latest consensus state: current_term: 1 leader_uuid: "d1cb67196292491b8760ec4c2c04e99b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cb67196292491b8760ec4c2c04e99b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33545 } } }
I20260620 21:59:01.249270 10737 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1cb67196292491b8760ec4c2c04e99b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cb67196292491b8760ec4c2c04e99b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33545 } } }
I20260620 21:59:01.249477 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 10665
I20260620 21:59:01.249459 10737 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:01.249451 10738 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:01.249583 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:01.249790 7110 external_mini_cluster.cc:1428] Setting key ed818a796f34a156b1f241a359458a80
I20260620 21:59:01.249938 10741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:01.250542 10741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:01.252672 10741 catalog_manager.cc:1347] Generated new cluster ID: d3f47dda23774e8083cf6aafa39d7620
I20260620 21:59:01.252730 10741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:01.261037 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.250508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.6.241.190:33545)
0620 21:59:01.250986 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:01.250998 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:01.251032 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:59:01.251271 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:01.251277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:01.251362 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:01.251523 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:01.252308 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.253390 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.254236 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.254811 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.254937 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:01.257386 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:01.257391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:01.257392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:01.257405 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:01.259304 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.259726 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.259730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.259730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.259851 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.260087 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.260088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.260088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.260366 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:01.260547 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:01.260651 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 21:59:01.260811 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":93,"threads_started":1}
I20260620 21:59:01.283418 10741 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:01.284025 10741 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:01.289980 10741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1cb67196292491b8760ec4c2c04e99b: Generated new TSK 0
I20260620 21:59:01.290203 10741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:59:01.321081 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33545
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:01.422443 10759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:01.422698 10759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:01.422725 10759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:01.423925 10759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:01.423972 10759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:01.424028 10759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:01.425573 10759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:01.425836 10759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:01.426079 10759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:01.431449 10759 server_base.cc:1034] running on GCE node
W20260620 21:59:01.431418 10765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.431411 10767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.431394 10764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:01.431882 10759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:01.432127 10759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:01.433295 10759 hybrid_clock.cc:648] HybridClock initialized: now 1781992741433281 us; error 32 us; skew 500 ppm
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:01.436005 10759 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:59:01.436483 10759 webserver.cc:460] Webserver started at http://127.6.241.129:39571/ using document root <none> and password file <none>
I20260620 21:59:01.436717 10759 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:01.436789 10759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:01.436947 10759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:01.438055 10759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "d7bf705326444e95b3f24d06b0585c8a"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "e61403097c0d7d517221e08b28a20138"
server_key_iv: "9cd4cc1a3be7a2f027d800a61ea28740"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.438436 10759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "d7bf705326444e95b3f24d06b0585c8a"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "e61403097c0d7d517221e08b28a20138"
server_key_iv: "9cd4cc1a3be7a2f027d800a61ea28740"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.439880 10759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:01.440691 10774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.440868 10759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 21:59:01.440969 10759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "d7bf705326444e95b3f24d06b0585c8a"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "e61403097c0d7d517221e08b28a20138"
server_key_iv: "9cd4cc1a3be7a2f027d800a61ea28740"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.441067 10759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:01.467555 10759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:01.469538 10759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:01.469719 10759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:01.470005 10759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:01.470332 10759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:01.470389 10759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.470431 10759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:01.470460 10759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.476414 10759 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:45659
I20260620 21:59:01.476491 10887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:45659 every 8 connection(s)
I20260620 21:59:01.476853 10759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:01.477352 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 10759
I20260620 21:59:01.477454 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:01.477674 7110 external_mini_cluster.cc:1428] Setting key cc3e29235627577b580bcaa102882b12
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
I20260620 21:59:01.487372 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.477683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:55917 (local address 127.6.241.190:33545)
0620 21:59:01.477731 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:01.477733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:01.478023 (+ 290us) server_negotiation.cc:407] Connection header received
0620 21:59:01.479182 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:01.479183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:01.479199 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:01.479251 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:01.479941 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.480463 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.481495 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.481728 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.481814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:01.484196 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:01.484198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:01.484198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:01.484216 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:01.485679 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.486125 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.486126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.486126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.486188 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.486713 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.486714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.486714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.486836 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:01.486901 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:01.487225 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 21:59:01.487304 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
I20260620 21:59:01.487572 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.477679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33545 (local address 127.6.241.129:55917)
0620 21:59:01.477864 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.477919 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:01.479068 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:01.479344 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:01.479348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:01.479629 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:01.479832 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:01.479836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.480563 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.480564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:01.481395 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:01.481397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.481826 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.481827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:01.481960 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:01.482557 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:01.482565 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:01.484089 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:01.485786 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:01.485794 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:01.485796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:01.486036 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:01.486451 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:01.486452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:01.486453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:01.486562 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:01.486903 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:01.486908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:01.487069 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 21:59:01.487176 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:59:01.487308 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260620 21:59:01.488119 10888 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33545
I20260620 21:59:01.488229 10888 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:01.488427 10888 heartbeater.cc:511] Master 127.6.241.190:33545 requested a full tablet report, sending...
I20260620 21:59:01.489117 10697 ts_manager.cc:194] Registered new tserver with Master: d7bf705326444e95b3f24d06b0585c8a (127.6.241.129:45659)
I20260620 21:59:01.489755 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:55917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:59:01.501159 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33545
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:01.586903 10893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:01.587193 10893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:01.587229 10893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:01.589071 10893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:01.589139 10893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:01.589180 10893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:01.591969 10893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:01.592425 10893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:01.592730 10893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:01.598295 10899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.598489 10901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.598556 10898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:01.598694 10893 server_base.cc:1034] running on GCE node
I20260620 21:59:01.598891 10893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:01.599157 10893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:01.600312 10893 hybrid_clock.cc:648] HybridClock initialized: now 1781992741600283 us; error 41 us; skew 500 ppm
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:01.603147 10893 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:59:01.603603 10893 webserver.cc:460] Webserver started at http://127.6.241.130:45843/ using document root <none> and password file <none>
I20260620 21:59:01.603888 10893 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:01.603964 10893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:01.604161 10893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:01.605803 10893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "71f4e65640734d2085efeaffcdb3f16d"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "15b9cb90d64d5e0cdf6b46990cac1d15"
server_key_iv: "7d600c0fff6427f70a482eddb753a167"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.606267 10893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "71f4e65640734d2085efeaffcdb3f16d"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "15b9cb90d64d5e0cdf6b46990cac1d15"
server_key_iv: "7d600c0fff6427f70a482eddb753a167"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.608173 10893 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:01.609215 10908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.609404 10893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:01.609488 10893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "71f4e65640734d2085efeaffcdb3f16d"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "15b9cb90d64d5e0cdf6b46990cac1d15"
server_key_iv: "7d600c0fff6427f70a482eddb753a167"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.609543 10893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:01.636911 10893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:01.639410 10893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:01.639541 10893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:01.639744 10893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:01.640092 10893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:01.640127 10893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.640177 10893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:01.640203 10893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.646440 10893 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:39533
I20260620 21:59:01.646574 11021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:39533 every 8 connection(s)
I20260620 21:59:01.646903 10893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:01.656596 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 10893
I20260620 21:59:01.656718 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:01.656733 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.647669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:35493 (local address 127.6.241.190:33545)
0620 21:59:01.647734 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:01.647736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:01.647964 (+ 228us) server_negotiation.cc:407] Connection header received
0620 21:59:01.648965 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:01.648966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:01.648977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:01.649045 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:01.649863 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.650402 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.651524 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.651711 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.651765 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:01.653849 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:01.653851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:01.653851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:01.653857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:01.655385 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.655790 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.655791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.655791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.655853 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.656088 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.656088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.656089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.656195 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:01.656246 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:01.656525 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 21:59:01.656620 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 21:59:01.656889 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.647652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33545 (local address 127.6.241.130:35493)
0620 21:59:01.647826 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.647869 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:01.648871 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:01.649153 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:01.649159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:01.649482 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:01.649782 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:01.649789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.650542 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.650543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:01.651434 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:01.651437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.651785 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.651786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:01.651885 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:01.652397 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:01.652405 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:01.653761 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:01.655465 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:01.655482 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:01.655484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:01.655720 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:01.655912 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:01.655913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:01.655913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:01.656027 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:01.656251 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:01.656255 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:01.656389 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 21:59:01.656482 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:59:01.656605 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260620 21:59:01.656906 7110 external_mini_cluster.cc:1428] Setting key 3f93e1bafc677426f5416cb32686373f
I20260620 21:59:01.657382 11022 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33545
I20260620 21:59:01.657492 11022 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:01.657691 11022 heartbeater.cc:511] Master 127.6.241.190:33545 requested a full tablet report, sending...
I20260620 21:59:01.658080 10697 ts_manager.cc:194] Registered new tserver with Master: 71f4e65640734d2085efeaffcdb3f16d (127.6.241.130:39533)
I20260620 21:59:01.658469 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:35493
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:59:01.682956 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33545
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:01.766232 11027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:01.766445 11027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:01.766470 11027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:01.767732 11027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:01.767782 11027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:01.767810 11027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:01.769379 11027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:01.769626 11027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:01.769869 11027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:01.774961 11032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.774961 11033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:01.775101 11027 server_base.cc:1034] running on GCE node
W20260620 21:59:01.775027 11035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:01.775350 11027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:01.775576 11027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:01.776779 11027 hybrid_clock.cc:648] HybridClock initialized: now 1781992741776765 us; error 42 us; skew 500 ppm
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:01.779302 11027 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:59:01.779798 11027 webserver.cc:460] Webserver started at http://127.6.241.131:39675/ using document root <none> and password file <none>
I20260620 21:59:01.780045 11027 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:01.780112 11027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:01.780258 11027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:01.781370 11027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "2502aec270c149c3a7904a4744d3a8d7"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "88b83de536f507c77d9cb19a53fc85ef"
server_key_iv: "8ddd19d2424471f8444832a51ae5a54b"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.781749 11027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "2502aec270c149c3a7904a4744d3a8d7"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "88b83de536f507c77d9cb19a53fc85ef"
server_key_iv: "8ddd19d2424471f8444832a51ae5a54b"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.783215 11027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:01.783982 11042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.784207 11027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:01.784313 11027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "2502aec270c149c3a7904a4744d3a8d7"
format_stamp: "Formatted at 2026-06-20 21:59:01 on dist-test-slave-gl2g"
server_key: "88b83de536f507c77d9cb19a53fc85ef"
server_key_iv: "8ddd19d2424471f8444832a51ae5a54b"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.784389 11027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:01.797608 11027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:01.799602 11027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:01.799762 11027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:01.800014 11027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:01.800328 11027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:01.800381 11027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.800426 11027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:01.800458 11027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.806622 11027 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:38755
I20260620 21:59:01.806668 11155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:38755 every 8 connection(s)
I20260620 21:59:01.806994 11027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:01.808085 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 11027
I20260620 21:59:01.808158 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:01.808339 7110 external_mini_cluster.cc:1428] Setting key a29217cf1cdf2ded57b69bb079d6afc5
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:01.816586 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.807712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:44231 (local address 127.6.241.190:33545)
0620 21:59:01.807772 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:01.807774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:01.807955 (+ 181us) server_negotiation.cc:407] Connection header received
0620 21:59:01.809177 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:01.809178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:01.809191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:01.809235 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:01.809850 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.810339 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.811364 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.811559 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.811613 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:01.813786 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:01.813788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:01.813788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:01.813795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:01.815263 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.815812 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.815813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.815814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.815888 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.816105 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.816106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.816106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.816211 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:01.816260 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:01.816451 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 21:59:01.816521 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 21:59:01.816782 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.807673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33545 (local address 127.6.241.131:44231)
0620 21:59:01.807831 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.807872 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:01.809073 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:01.809303 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:01.809307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:01.809579 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:01.809771 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:01.809775 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.810416 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.810417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:01.811293 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:01.811295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.811632 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.811633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:01.811733 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:01.812296 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:01.812305 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:01.813704 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:01.815347 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:01.815355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:01.815358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:01.815706 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:01.815946 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:01.815947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:01.815947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:01.816051 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:01.816251 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:01.816254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:01.816365 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 21:59:01.816439 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 21:59:01.816573 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260620 21:59:01.817332 11156 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33545
I20260620 21:59:01.817440 11156 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:01.817675 11156 heartbeater.cc:511] Master 127.6.241.190:33545 requested a full tablet report, sending...
I20260620 21:59:01.818101 10697 ts_manager.cc:194] Registered new tserver with Master: 2502aec270c149c3a7904a4744d3a8d7 (127.6.241.131:38755)
I20260620 21:59:01.818488 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:44231
I20260620 21:59:01.819938 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:59:01.843581 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.835386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.6.241.129:45659)
0620 21:59:01.835588 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:01.835593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:01.835604 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:59:01.835630 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:01.835632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:01.835646 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:01.835714 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:01.836345 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.837080 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.837815 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.838165 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.838220 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:01.840638 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:01.840648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:01.840648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:01.840655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:01.842281 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.842736 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.842740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.842740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.842802 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.843007 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.843008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.843009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.843174 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:01.843308 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:01.843396 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 21:59:01.843461 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:01.851490 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.844931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.6.241.190:33545)
0620 21:59:01.844975 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:01.844977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:01.845006 (+ 29us) server_negotiation.cc:407] Connection header received
0620 21:59:01.845067 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:01.845068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:01.845076 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:01.845124 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:01.845573 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.846044 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.846763 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:01.846938 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:01.846992 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:01.849127 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:01.849128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:01.849128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:01.849135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:01.850453 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.850810 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.850811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.850812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.850860 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:01.851036 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:01.851037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:01.851037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:01.851168 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:01.851274 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:01.851351 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 21:59:01.851411 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 21:59:01.851709 10697 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35852
I20260620 21:59:01.852097 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 10759
I20260620 21:59:01.857110 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 10893
I20260620 21:59:01.861861 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 11027
I20260620 21:59:01.866050 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 10665
2026-06-20T21:59:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1921 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[11168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[11168](info): set up 2 sockets
Jun 20 21:59:01 dist-test-slave-gl2g krb5kdc[11168](info): commencing operation
krb5kdc: starting...
W20260620 21:59:02.898783 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:59:02 dist-test-slave-gl2g krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:59:03.018170 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40655
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:03.101295 11181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:03.101536 11181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:03.101562 11181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:03.102790 11181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:03.102835 11181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:03.102850 11181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:03.102864 11181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:03.102876 11181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:03.104417 11181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:40655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:03.104651 11181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:03.104831 11181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:03.109653 11186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:03.109671 11187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:03.109720 11189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:03.109875 11181 server_base.cc:1034] running on GCE node
I20260620 21:59:03.110140 11181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:03.110438 11181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:03.111582 11181 hybrid_clock.cc:648] HybridClock initialized: now 1781992743111554 us; error 40 us; skew 500 ppm
Jun 20 21:59:03 dist-test-slave-gl2g krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992743, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:03.114202 11181 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:59:03.114558 11181 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:03.114661 11181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:03.114814 11181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:03.115967 11181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "450224d49f994d37aea60deb9eb049e6"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "6e538b5e6ebe13eb401c5e3510f383ec"
server_key_iv: "a4a7798fabe00027303bfaba9ee97919"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.116353 11181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "450224d49f994d37aea60deb9eb049e6"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "6e538b5e6ebe13eb401c5e3510f383ec"
server_key_iv: "a4a7798fabe00027303bfaba9ee97919"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.117660 11181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 21:59:03.118396 11195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.118569 11181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 21:59:03.118702 11181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "450224d49f994d37aea60deb9eb049e6"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "6e538b5e6ebe13eb401c5e3510f383ec"
server_key_iv: "a4a7798fabe00027303bfaba9ee97919"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.118780 11181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:03.134411 11181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:03.136580 11181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:03.136758 11181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:03.140781 11181 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:40655
I20260620 21:59:03.140825 11247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:40655 every 8 connection(s)
I20260620 21:59:03.141157 11181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:03.142163 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:03.143112 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 11181
I20260620 21:59:03.143213 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:03.143397 7110 external_mini_cluster.cc:1428] Setting key 4479a174449439c16a36741f3ad9a9c6
I20260620 21:59:03.145582 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6: Bootstrap starting.
I20260620 21:59:03.146216 11248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:03.146504 11248 log.cc:824] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:03.147253 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6: No bootstrap required, opened a new log
I20260620 21:59:03.149756 11248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450224d49f994d37aea60deb9eb049e6" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40655 } }
I20260620 21:59:03.149940 11248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:03.149968 11248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450224d49f994d37aea60deb9eb049e6, State: Initialized, Role: FOLLOWER
I20260620 21:59:03.150120 11248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450224d49f994d37aea60deb9eb049e6" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40655 } }
I20260620 21:59:03.150198 11248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:03.150241 11248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:03.150288 11248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 21:59:03 dist-test-slave-gl2g krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:03.150934 11248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450224d49f994d37aea60deb9eb049e6" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40655 } }
I20260620 21:59:03.151059 11248 leader_election.cc:302] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 450224d49f994d37aea60deb9eb049e6; no voters:
I20260620 21:59:03.151259 11248 leader_election.cc:288] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:03.151289 11253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:03.151397 11253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [term 1 LEADER]: Becoming Leader. State: Replica: 450224d49f994d37aea60deb9eb049e6, State: Running, Role: LEADER
I20260620 21:59:03.151484 11253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450224d49f994d37aea60deb9eb049e6" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40655 } }
I20260620 21:59:03.151531 11248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:03.151774 11253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 450224d49f994d37aea60deb9eb049e6. Latest consensus state: current_term: 1 leader_uuid: "450224d49f994d37aea60deb9eb049e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450224d49f994d37aea60deb9eb049e6" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40655 } } }
I20260620 21:59:03.151841 11253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:03.152078 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "450224d49f994d37aea60deb9eb049e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450224d49f994d37aea60deb9eb049e6" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40655 } } }
I20260620 21:59:03.152153 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:03.152401 11257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:03.152976 11257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:03.154747 11257 catalog_manager.cc:1347] Generated new cluster ID: d7d43b19d9f84bf9b191ddf06d07e203
I20260620 21:59:03.154795 11257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:03.155478 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.143942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.6.241.190:40655)
0620 21:59:03.144132 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:03.144146 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:03.144172 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:59:03.144605 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:03.144611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:03.144659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:03.144773 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:03.145583 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.146655 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.147676 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.148072 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.148162 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.151069 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:03.151073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:03.151074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:03.151086 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:03.153406 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.154121 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.154123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.154124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.154201 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.154419 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.154421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.154422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.154753 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:03.154905 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:03.155160 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 21:59:03.155290 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260620 21:59:03.178218 11257 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:03.179073 11257 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:03.196501 11257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6: Generated new TSK 0
I20260620 21:59:03.196753 11257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:59:03.226150 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40655
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:03.312157 11274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:03.312384 11274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:03.312412 11274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:03.313618 11274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:03.313665 11274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:03.313692 11274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:03.315596 11274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:03.315900 11274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:03.316155 11274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:03.321303 11279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:03.321298 11280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:03.321342 11274 server_base.cc:1034] running on GCE node
W20260620 21:59:03.321303 11282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:03.321835 11274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:03.322106 11274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:03.323269 11274 hybrid_clock.cc:648] HybridClock initialized: now 1781992743323265 us; error 35 us; skew 500 ppm
Jun 20 21:59:03 dist-test-slave-gl2g krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992743, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:03.325781 11274 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:59:03.326146 11274 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:03.326221 11274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:03.326381 11274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:03.327553 11274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "e9a89155afce4b3e9e1107010d9f98d8"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "9b30510287b78025a75e5fd57a58f408"
server_key_iv: "47aba35b9bf1de5e92c092908a8fe780"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.327925 11274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "e9a89155afce4b3e9e1107010d9f98d8"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "9b30510287b78025a75e5fd57a58f408"
server_key_iv: "47aba35b9bf1de5e92c092908a8fe780"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.329245 11274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:03.330015 11288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.330327 11274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:03.330430 11274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "e9a89155afce4b3e9e1107010d9f98d8"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "9b30510287b78025a75e5fd57a58f408"
server_key_iv: "47aba35b9bf1de5e92c092908a8fe780"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.330518 11274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:03.352464 11274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:03.354350 11274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:03.354532 11274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:03.354791 11274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:03.355136 11274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:03.355193 11274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.355234 11274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:03.355263 11274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.361214 11274 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:43465
I20260620 21:59:03.361250 11401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:43465 every 8 connection(s)
I20260620 21:59:03.361588 11274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:59:03 dist-test-slave-gl2g krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992743, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:03.370718 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.362380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:57217 (local address 127.6.241.190:40655)
0620 21:59:03.362459 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:03.362461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:03.362696 (+ 235us) server_negotiation.cc:407] Connection header received
0620 21:59:03.363199 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:03.363200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:03.363209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:03.363267 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:03.364222 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.364720 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.365701 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.365903 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.365965 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.368175 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:03.368177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:03.368177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:03.368184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:03.369399 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.369770 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.369771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.369771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.369823 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.370076 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.370077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.370077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.370200 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:03.370269 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:03.370498 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 21:59:03.370578 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:59:03.370878 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.362354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40655 (local address 127.6.241.129:57217)
0620 21:59:03.362550 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:03.362618 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:03.363117 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:03.363332 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:03.363335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:03.363604 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:03.364130 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.364135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.364802 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.364802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.365617 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.365619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.365979 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.365979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.366091 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.366694 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:03.366703 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:03.368083 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:03.369468 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:03.369474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:03.369476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:03.369703 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:03.369881 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:03.369881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:03.369882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:03.369990 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:03.370266 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:03.370270 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:03.370386 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 21:59:03.370466 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:59:03.370650 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":115,"threads_started":1}
I20260620 21:59:03.371346 11402 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40655
I20260620 21:59:03.371429 11402 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:03.371606 11402 heartbeater.cc:511] Master 127.6.241.190:40655 requested a full tablet report, sending...
I20260620 21:59:03.371657 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 11274
I20260620 21:59:03.371747 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:03.371982 7110 external_mini_cluster.cc:1428] Setting key b11a7b28ad9daa0f8d7475ff5072de22
I20260620 21:59:03.372213 11212 ts_manager.cc:194] Registered new tserver with Master: e9a89155afce4b3e9e1107010d9f98d8 (127.6.241.129:43465)
I20260620 21:59:03.373015 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:57217
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:59:03.397198 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40655
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:03.479277 11407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:03.479547 11407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:03.479588 11407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:03.480791 11407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:03.480883 11407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:03.480930 11407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:03.482496 11407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:03.482812 11407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:03.483063 11407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:03.488004 11413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:03.488021 11412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:03.488021 11415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:03.488313 11407 server_base.cc:1034] running on GCE node
I20260620 21:59:03.488549 11407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:03.488854 11407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:03.490038 11407 hybrid_clock.cc:648] HybridClock initialized: now 1781992743490013 us; error 38 us; skew 500 ppm
Jun 20 21:59:03 dist-test-slave-gl2g krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992743, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:03.492722 11407 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:59:03.493088 11407 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:03.493137 11407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:03.493232 11407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:03.494264 11407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "070abcd530004028b3447ed377eb144b"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "10e31b2bc6405bd3556efbe0e0db582d"
server_key_iv: "74b87e7a4ba83af09d046e46242f3690"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.494566 11407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "070abcd530004028b3447ed377eb144b"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "10e31b2bc6405bd3556efbe0e0db582d"
server_key_iv: "74b87e7a4ba83af09d046e46242f3690"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.495949 11407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:03.496709 11421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.496894 11407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 21:59:03.496968 11407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "070abcd530004028b3447ed377eb144b"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "10e31b2bc6405bd3556efbe0e0db582d"
server_key_iv: "74b87e7a4ba83af09d046e46242f3690"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.497049 11407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:03.515744 11407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:03.517818 11407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:03.517937 11407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:03.518155 11407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:03.518507 11407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:03.518543 11407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.518594 11407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:03.518646 11407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.525162 11407 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:36969
I20260620 21:59:03.525193 11534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:36969 every 8 connection(s)
I20260620 21:59:03.525497 11407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
Jun 20 21:59:03 dist-test-slave-gl2g krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992743, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:03.532326 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 11407
I20260620 21:59:03.532438 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:03.532660 7110 external_mini_cluster.cc:1428] Setting key 3ac93101ec6a71f97f44d1cacaf17207
I20260620 21:59:03.535689 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.526230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:38185 (local address 127.6.241.190:40655)
0620 21:59:03.526306 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:03.526309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:03.526561 (+ 252us) server_negotiation.cc:407] Connection header received
0620 21:59:03.527132 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:03.527133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:03.527144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:03.527211 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:03.528163 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.528648 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.529616 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.529821 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.529876 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.532285 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:03.532287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:03.532287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:03.532294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:03.534136 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.534689 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.534691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.534691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.534754 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.535017 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.535018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.535018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.535144 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:03.535219 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:03.535505 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 21:59:03.535598 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:59:03.535905 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.526211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40655 (local address 127.6.241.130:38185)
0620 21:59:03.526396 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:03.526433 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:03.527000 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:03.527288 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:03.527291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:03.527581 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:03.528083 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.528087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.528724 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.528725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.529541 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.529543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.529909 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.529910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.530068 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.530541 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:03.530550 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:03.532148 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:03.534263 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:03.534270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:03.534273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:03.534572 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:03.534821 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:03.534822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:03.534822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:03.534944 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:03.535208 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:03.535215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:03.535366 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 21:59:03.535472 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 21:59:03.535640 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
I20260620 21:59:03.536399 11535 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40655
I20260620 21:59:03.536515 11535 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:03.536684 11535 heartbeater.cc:511] Master 127.6.241.190:40655 requested a full tablet report, sending...
I20260620 21:59:03.537067 11212 ts_manager.cc:194] Registered new tserver with Master: 070abcd530004028b3447ed377eb144b (127.6.241.130:36969)
I20260620 21:59:03.537628 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:38185
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:59:03.557188 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40655
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:03.639602 11540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:03.639817 11540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:03.639842 11540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:03.640939 11540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:03.640973 11540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:03.641000 11540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:03.642532 11540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:03.642795 11540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:03.642987 11540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:03.647941 11546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:03.647897 11545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:03.647897 11548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:03.648106 11540 server_base.cc:1034] running on GCE node
I20260620 21:59:03.648429 11540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:03.648638 11540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:03.649777 11540 hybrid_clock.cc:648] HybridClock initialized: now 1781992743649764 us; error 29 us; skew 500 ppm
Jun 20 21:59:03 dist-test-slave-gl2g krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992743, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:03.652686 11540 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:59:03.653105 11540 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:03.653203 11540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:03.653359 11540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:03.654438 11540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "672acee82b22436834b2eaebfc177db5"
server_key_iv: "386021f691a318f851aeb5e7cf98462c"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.654830 11540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "672acee82b22436834b2eaebfc177db5"
server_key_iv: "386021f691a318f851aeb5e7cf98462c"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.656155 11540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:03.656953 11554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.657195 11540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:03.657325 11540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9"
format_stamp: "Formatted at 2026-06-20 21:59:03 on dist-test-slave-gl2g"
server_key: "672acee82b22436834b2eaebfc177db5"
server_key_iv: "386021f691a318f851aeb5e7cf98462c"
server_key_version: "encryptionkey@0"
I20260620 21:59:03.657413 11540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:03.672276 11540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:03.674289 11540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:03.674464 11540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:03.674769 11540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:03.675135 11540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:03.675192 11540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.675233 11540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:03.675261 11540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:03.681227 11540 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:45567
I20260620 21:59:03.681257 11667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:45567 every 8 connection(s)
I20260620 21:59:03.681546 11540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:03.682144 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 11540
I20260620 21:59:03.682260 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:03.682478 7110 external_mini_cluster.cc:1428] Setting key 4d00e4c2010869421e98c0c1d63d579f
Jun 20 21:59:03 dist-test-slave-gl2g krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992743, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:03.691547 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.682297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:42841 (local address 127.6.241.190:40655)
0620 21:59:03.682349 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:03.682351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:03.682633 (+ 282us) server_negotiation.cc:407] Connection header received
0620 21:59:03.683372 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:03.683373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:03.683383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:03.683428 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:03.684452 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.685038 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.686156 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.686351 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.686419 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.688772 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:03.688773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:03.688774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:03.688781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:03.690194 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.690660 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.690662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.690662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.690727 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.690983 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.690984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.690984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.691100 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:03.691154 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:03.691403 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 21:59:03.691478 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:59:03.691784 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.682277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40655 (local address 127.6.241.131:42841)
0620 21:59:03.682447 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:03.682501 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:03.683259 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:03.683488 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:03.683492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:03.683765 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:03.684285 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.684291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.685160 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.685162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.686045 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.686047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.686465 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.686466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.686669 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.687207 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:03.687216 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:03.688651 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:03.690277 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:03.690283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:03.690285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:03.690532 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:03.690788 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:03.690789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:03.690789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:03.690899 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:03.691158 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:03.691163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:03.691289 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 21:59:03.691379 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:59:03.691514 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260620 21:59:03.692411 11668 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40655
I20260620 21:59:03.692634 11668 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:03.692834 11668 heartbeater.cc:511] Master 127.6.241.190:40655 requested a full tablet report, sending...
I20260620 21:59:03.693269 11212 ts_manager.cc:194] Registered new tserver with Master: c4c4b867cd4e4fe4baf4a59738ed57d9 (127.6.241.131:45567)
I20260620 21:59:03.693763 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:42841
I20260620 21:59:03.693920 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:03.701556 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.695302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.6.241.190:40655)
0620 21:59:03.695375 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:03.695378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:03.695527 (+ 149us) server_negotiation.cc:407] Connection header received
0620 21:59:03.695550 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:03.695551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:03.695558 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:03.695607 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:03.696138 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.696707 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.697416 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.697790 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.697864 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.699226 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:03.699228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:03.699228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:03.699237 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:03.700546 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.700868 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.700869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.700870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.700933 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:03.701116 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:03.701117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:03.701117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:03.701260 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:03.701338 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:03.701421 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 21:59:03.701478 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:59:03.703334 11212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:03.704958 11212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:03.712088 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.707614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.6.241.131:45567)
0620 21:59:03.707965 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:03.707971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:03.707981 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:03.708023 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:03.708025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:03.708042 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:03.708325 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:03.708434 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.709554 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.711067 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.711627 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.711714 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.711770 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:03.711944 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 21:59:03.711991 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":89,"threads_started":1}
I20260620 21:59:03.712076 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.707528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:45567 (local address 127.0.0.1:58718)
0620 21:59:03.707692 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:03.707734 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:03.707798 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:03.708155 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:03.708158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:03.708161 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:03.708242 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.708244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.709660 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.709661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.710987 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.710989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.711726 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.711727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.711810 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.711825 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 21:59:03.711926 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 21:59:03.711999 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":20,"thread_start_us":58,"threads_started":1}
I20260620 21:59:03.712119 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.707785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.6.241.130:36969)
0620 21:59:03.708080 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:03.708083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:03.708093 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:03.708281 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:03.708284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:03.708302 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:03.708377 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:03.708595 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.709684 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.710809 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.711626 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.711702 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.711763 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:03.711988 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 21:59:03.712030 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":83,"threads_started":1}
I20260620 21:59:03.712119 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.707789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.6.241.129:43465)
0620 21:59:03.707993 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:03.707997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:03.708175 (+ 178us) server_negotiation.cc:407] Connection header received
0620 21:59:03.708244 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:03.708247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:03.708263 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:03.708338 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:03.708584 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.709270 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.710853 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.711642 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.711702 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.711764 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:03.711988 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 21:59:03.712031 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20260620 21:59:03.712374 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.707548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:43465 (local address 127.0.0.1:55112)
0620 21:59:03.708120 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:03.708129 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:03.708174 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:03.708436 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:03.708437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:03.708439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:03.708488 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.708491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.709368 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.709370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.710786 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.710789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.711800 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.711801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.711884 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.711886 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:59:03.711936 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 21:59:03.712283 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":334,"thread_start_us":92,"threads_started":1}
I20260620 21:59:03.712373 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.707546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:36969 (local address 127.0.0.1:47084)
0620 21:59:03.707959 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:03.707968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:03.708012 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:03.708470 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:03.708471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:03.708473 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:03.708520 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.708522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.709789 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.709791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.710726 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:03.710729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:03.711799 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:03.711800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:03.711881 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:03.711885 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:03.711943 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 21:59:03.712282 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":300,"thread_start_us":90,"threads_started":1}
I20260620 21:59:03.713204 11336 tablet_service.cc:1467] Processing CreateTablet for tablet 092514765a7443eb86dde6df30b283ea (DEFAULT_TABLE table=test-table [id=8abb528d85a84fa6b80be9e6de195fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:03.713287 11469 tablet_service.cc:1467] Processing CreateTablet for tablet 092514765a7443eb86dde6df30b283ea (DEFAULT_TABLE table=test-table [id=8abb528d85a84fa6b80be9e6de195fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:03.713467 11602 tablet_service.cc:1467] Processing CreateTablet for tablet 092514765a7443eb86dde6df30b283ea (DEFAULT_TABLE table=test-table [id=8abb528d85a84fa6b80be9e6de195fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:03.713608 11336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092514765a7443eb86dde6df30b283ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:03.713634 11469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092514765a7443eb86dde6df30b283ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:03.713727 11602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092514765a7443eb86dde6df30b283ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:03.716955 11685 tablet_bootstrap.cc:492] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8: Bootstrap starting.
I20260620 21:59:03.716936 11686 tablet_bootstrap.cc:492] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: Bootstrap starting.
I20260620 21:59:03.717546 11685 tablet_bootstrap.cc:654] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:03.717558 11686 tablet_bootstrap.cc:654] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:03.718082 11685 log.cc:824] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:03.718081 11686 log.cc:824] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:03.718474 11687 tablet_bootstrap.cc:492] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: Bootstrap starting.
I20260620 21:59:03.718815 11685 tablet_bootstrap.cc:492] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8: No bootstrap required, opened a new log
I20260620 21:59:03.718894 11685 ts_tablet_manager.cc:1397] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:03.719091 11687 tablet_bootstrap.cc:654] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:03.719595 11687 log.cc:824] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:03.720116 11686 tablet_bootstrap.cc:492] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: No bootstrap required, opened a new log
I20260620 21:59:03.720172 11686 ts_tablet_manager.cc:1397] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:59:03.720477 11685 raft_consensus.cc:348] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } }
I20260620 21:59:03.720639 11685 raft_consensus.cc:374] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:03.720685 11685 raft_consensus.cc:729] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a89155afce4b3e9e1107010d9f98d8, State: Initialized, Role: FOLLOWER
I20260620 21:59:03.720841 11685 consensus_queue.cc:260] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } }
I20260620 21:59:03.721168 11685 ts_tablet_manager.cc:1428] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:03.721203 11402 heartbeater.cc:503] Master 127.6.241.190:40655 was elected leader, sending a full tablet report...
I20260620 21:59:03.722019 11686 raft_consensus.cc:348] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } }
I20260620 21:59:03.722231 11686 raft_consensus.cc:374] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:03.722280 11686 raft_consensus.cc:729] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070abcd530004028b3447ed377eb144b, State: Initialized, Role: FOLLOWER
I20260620 21:59:03.722398 11686 consensus_queue.cc:260] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } }
I20260620 21:59:03.722733 11686 ts_tablet_manager.cc:1428] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 21:59:03.722783 11535 heartbeater.cc:503] Master 127.6.241.190:40655 was elected leader, sending a full tablet report...
I20260620 21:59:03.723569 11687 tablet_bootstrap.cc:492] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: No bootstrap required, opened a new log
I20260620 21:59:03.723664 11687 ts_tablet_manager.cc:1397] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 21:59:03.725217 11687 raft_consensus.cc:348] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } }
I20260620 21:59:03.725392 11687 raft_consensus.cc:374] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:03.725446 11687 raft_consensus.cc:729] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c4b867cd4e4fe4baf4a59738ed57d9, State: Initialized, Role: FOLLOWER
I20260620 21:59:03.725584 11687 consensus_queue.cc:260] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } }
I20260620 21:59:03.725893 11687 ts_tablet_manager.cc:1428] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:03.725912 11668 heartbeater.cc:503] Master 127.6.241.190:40655 was elected leader, sending a full tablet report...
W20260620 21:59:03.776166 11536 tablet.cc:2369] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:59:03.862298 11403 tablet.cc:2369] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:59:03.932150 11669 tablet.cc:2369] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:04.093801 11693 raft_consensus.cc:482] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:04.094009 11693 raft_consensus.cc:504] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } }
I20260620 21:59:04.094440 11693 leader_election.cc:288] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a89155afce4b3e9e1107010d9f98d8 (127.6.241.129:43465), 070abcd530004028b3447ed377eb144b (127.6.241.130:36969)
I20260620 21:59:04.097422 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.094742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:48171 (local address 127.6.241.130:36969)
0620 21:59:04.094787 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.094790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.094854 (+ 64us) server_negotiation.cc:407] Connection header received
0620 21:59:04.094957 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.094958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.094968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.095016 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:04.095215 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.095704 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.096693 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.097050 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.097146 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:04.097155 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.097312 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.097352 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:59:04.097422 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.094671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:36969 (local address 127.6.241.131:48171)
0620 21:59:04.094753 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.094767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:04.094857 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:04.095073 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:04.095074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:04.095076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:04.095153 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:04.095155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.095794 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.095795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:04.096581 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:04.096583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.097141 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.097142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:04.097230 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:04.097233 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:04.097304 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 21:59:04.097347 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 21:59:04.097499 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.094782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:50503 (local address 127.6.241.129:43465)
0620 21:59:04.094836 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.094838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.094979 (+ 141us) server_negotiation.cc:407] Connection header received
0620 21:59:04.095053 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.095061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.095071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.095126 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:04.095317 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.095786 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.096744 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.097123 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.097210 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:04.097219 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.097382 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.097427 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:59:04.097530 11694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.094686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:43465 (local address 127.6.241.131:50503)
0620 21:59:04.094888 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.094898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:04.094988 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:04.095153 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:04.095154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:04.095156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:04.095233 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:04.095235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.095893 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.095894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:04.096645 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:04.096647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.097209 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.097210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:04.097294 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:04.097297 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:04.097364 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 21:59:04.097407 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":101,"threads_started":1}
I20260620 21:59:04.097908 11356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092514765a7443eb86dde6df30b283ea" candidate_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a89155afce4b3e9e1107010d9f98d8" is_pre_election: true
I20260620 21:59:04.097913 11489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092514765a7443eb86dde6df30b283ea" candidate_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070abcd530004028b3447ed377eb144b" is_pre_election: true
I20260620 21:59:04.098084 11356 raft_consensus.cc:2454] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c4b867cd4e4fe4baf4a59738ed57d9 in term 0.
I20260620 21:59:04.098084 11489 raft_consensus.cc:2454] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c4b867cd4e4fe4baf4a59738ed57d9 in term 0.
I20260620 21:59:04.098351 11558 leader_election.cc:302] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4c4b867cd4e4fe4baf4a59738ed57d9, e9a89155afce4b3e9e1107010d9f98d8; no voters:
I20260620 21:59:04.098536 11693 raft_consensus.cc:2781] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:04.098620 11693 raft_consensus.cc:482] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:04.098640 11693 raft_consensus.cc:3037] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:04.099380 11693 raft_consensus.cc:504] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } }
I20260620 21:59:04.099545 11693 leader_election.cc:288] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [CANDIDATE]: Term 1 election: Requested vote from peers e9a89155afce4b3e9e1107010d9f98d8 (127.6.241.129:43465), 070abcd530004028b3447ed377eb144b (127.6.241.130:36969)
I20260620 21:59:04.099712 11356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092514765a7443eb86dde6df30b283ea" candidate_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a89155afce4b3e9e1107010d9f98d8"
I20260620 21:59:04.099727 11489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092514765a7443eb86dde6df30b283ea" candidate_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070abcd530004028b3447ed377eb144b"
I20260620 21:59:04.099798 11356 raft_consensus.cc:3037] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:04.099798 11489 raft_consensus.cc:3037] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:04.100677 11489 raft_consensus.cc:2454] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c4b867cd4e4fe4baf4a59738ed57d9 in term 1.
I20260620 21:59:04.100685 11356 raft_consensus.cc:2454] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c4b867cd4e4fe4baf4a59738ed57d9 in term 1.
I20260620 21:59:04.100870 11558 leader_election.cc:302] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4c4b867cd4e4fe4baf4a59738ed57d9, e9a89155afce4b3e9e1107010d9f98d8; no voters:
I20260620 21:59:04.100997 11693 raft_consensus.cc:2781] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:04.101171 11693 raft_consensus.cc:686] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 1 LEADER]: Becoming Leader. State: Replica: c4c4b867cd4e4fe4baf4a59738ed57d9, State: Running, Role: LEADER
I20260620 21:59:04.101284 11693 consensus_queue.cc:237] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } }
I20260620 21:59:04.102229 11212 catalog_manager.cc:5495] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4c4b867cd4e4fe4baf4a59738ed57d9 (127.6.241.131). New cstate: current_term: 1 leader_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c4b867cd4e4fe4baf4a59738ed57d9" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 } health_report { overall_health: UNKNOWN } } }
I20260620 21:59:04.152801 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.150068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.6.241.131:45567)
0620 21:59:04.150145 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.150147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.150156 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:04.150202 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.150203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.150220 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.150284 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:04.150415 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.150988 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.151863 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:04.152094 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:04.152163 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:04.152337 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:04.152540 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:04.152610 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.152649 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.152670 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:59:04.154779 11489 raft_consensus.cc:1261] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [term 1 FOLLOWER]: Refusing update from remote peer c4c4b867cd4e4fe4baf4a59738ed57d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:04.154824 11356 raft_consensus.cc:1261] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8 [term 1 FOLLOWER]: Refusing update from remote peer c4c4b867cd4e4fe4baf4a59738ed57d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:04.155197 11695 consensus_queue.cc:1035] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a89155afce4b3e9e1107010d9f98d8" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:04.155336 11693 consensus_queue.cc:1035] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070abcd530004028b3447ed377eb144b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:04.158211 11698 mvcc.cc:205] Tried to move back new op lower bound from 7299042280054624256 to 7299042279839473664. Current Snapshot: MvccSnapshot[applied={T|T < 7299042280054624256}]
I20260620 21:59:04.158885 11699 mvcc.cc:205] Tried to move back new op lower bound from 7299042280054624256 to 7299042279839473664. Current Snapshot: MvccSnapshot[applied={T|T < 7299042280054624256 or (T in {7299042280054624256})}]
I20260620 21:59:04.160785 11212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:04.160904 11212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:04.161759 11212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 450224d49f994d37aea60deb9eb049e6: Sending DeleteTablet for 3 replicas of tablet 092514765a7443eb86dde6df30b283ea
I20260620 21:59:04.162046 11602 tablet_service.cc:1514] Processing DeleteTablet for tablet 092514765a7443eb86dde6df30b283ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:04 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:58718
I20260620 21:59:04.162099 11336 tablet_service.cc:1514] Processing DeleteTablet for tablet 092514765a7443eb86dde6df30b283ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:04 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:55112
I20260620 21:59:04.162336 11708 tablet_replica.cc:330] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8: stopping tablet replica
I20260620 21:59:04.162473 11708 raft_consensus.cc:2229] T 092514765a7443eb86dde6df30b283ea P e9a89155afce4b3e9e1107010d9f98d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:04.162544 11469 tablet_service.cc:1514] Processing DeleteTablet for tablet 092514765a7443eb86dde6df30b283ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:04 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:47084
I20260620 21:59:04.162788 11709 tablet_replica.cc:330] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: stopping tablet replica
I20260620 21:59:04.162797 11707 tablet_replica.cc:330] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: stopping tablet replica
I20260620 21:59:04.162848 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 11274
I20260620 21:59:04.162889 11707 raft_consensus.cc:2229] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:04.162894 11709 raft_consensus.cc:2229] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:04.162998 11709 raft_consensus.cc:2258] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:04.164100 11709 ts_tablet_manager.cc:1905] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:04.164749 11707 raft_consensus.cc:2258] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:04.165205 11707 ts_tablet_manager.cc:1905] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:04.165887 11709 ts_tablet_manager.cc:1918] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:04.165971 11709 log.cc:1196] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/092514765a7443eb86dde6df30b283ea
I20260620 21:59:04.166244 11709 ts_tablet_manager.cc:1939] T 092514765a7443eb86dde6df30b283ea P 070abcd530004028b3447ed377eb144b: Deleting consensus metadata
I20260620 21:59:04.166700 11197 catalog_manager.cc:4841] TS 070abcd530004028b3447ed377eb144b (127.6.241.130:36969): tablet 092514765a7443eb86dde6df30b283ea (table test-table [id=8abb528d85a84fa6b80be9e6de195fa7]) successfully deleted
I20260620 21:59:04.167183 11707 ts_tablet_manager.cc:1918] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:04.167256 11707 log.cc:1196] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/092514765a7443eb86dde6df30b283ea
I20260620 21:59:04.167495 11707 ts_tablet_manager.cc:1939] T 092514765a7443eb86dde6df30b283ea P c4c4b867cd4e4fe4baf4a59738ed57d9: Deleting consensus metadata
I20260620 21:59:04.167908 11198 catalog_manager.cc:4841] TS c4c4b867cd4e4fe4baf4a59738ed57d9 (127.6.241.131:45567): tablet 092514765a7443eb86dde6df30b283ea (table test-table [id=8abb528d85a84fa6b80be9e6de195fa7]) successfully deleted
W20260620 21:59:04.169523 11199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:43465 (error 108)
I20260620 21:59:04.170075 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 11407
I20260620 21:59:04.170143 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.170000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:43465 (local address 127.0.0.1:55126)
0620 21:59:04.170066 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.170110 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:43465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:59:04.170992 11199 catalog_manager.cc:4584] TS e9a89155afce4b3e9e1107010d9f98d8 (127.6.241.129:43465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 092514765a7443eb86dde6df30b283ea: Network error: Client connection negotiation failed: client connection to 127.6.241.129:43465: connect: Connection refused (error 111)
I20260620 21:59:04.175740 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 11540
I20260620 21:59:04.180205 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 11181
2026-06-20T21:59:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2313 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T21:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:04Z Disabled control of system clock
I20260620 21:59:04.207747 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42247
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:44643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:59:04.290439 11714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:04.290719 11714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:04.290743 11714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:04.292093 11714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:04.292145 11714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:04.292158 11714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:04.292171 11714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:04.292183 11714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:04.293653 11714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:42247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:04.293946 11714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:04.294273 11714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:04.299466 11719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:04.299461 11720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:04.299461 11722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:04.299682 11714 server_base.cc:1034] running on GCE node
I20260620 21:59:04.300005 11714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:04.300344 11714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:04.301517 11714 hybrid_clock.cc:648] HybridClock initialized: now 1781992744301511 us; error 38 us; skew 500 ppm
I20260620 21:59:04.302899 11714 webserver.cc:460] Webserver started at http://127.6.241.190:44721/ using document root <none> and password file <none>
I20260620 21:59:04.303161 11714 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:04.303236 11714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:04.303386 11714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:04.304545 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "8a8cf0f579634c48a3413d224c10c325"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "f66277d4b46e448e632f62556559e1fe"
server_key_iv: "f6a50d3d114f2a0af154122adef66e76"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.304929 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "8a8cf0f579634c48a3413d224c10c325"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "f66277d4b46e448e632f62556559e1fe"
server_key_iv: "f6a50d3d114f2a0af154122adef66e76"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.306392 11714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:04.307263 11728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.307439 11714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:04.307544 11714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "8a8cf0f579634c48a3413d224c10c325"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "f66277d4b46e448e632f62556559e1fe"
server_key_iv: "f6a50d3d114f2a0af154122adef66e76"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.307633 11714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:04.311177 11714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:04.311470 11714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:04.311620 11714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:04.315796 11714 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:42247
I20260620 21:59:04.315826 11780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:42247 every 8 connection(s)
I20260620 21:59:04.316211 11714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:04.316896 11781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:04.319918 11781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325: Bootstrap starting.
I20260620 21:59:04.320581 11781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:04.320894 11781 log.cc:824] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:04.321640 11781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325: No bootstrap required, opened a new log
I20260620 21:59:04.322077 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 11714
I20260620 21:59:04.322161 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:04.322348 7110 external_mini_cluster.cc:1428] Setting key dc485dfe9e446ea44905487f4f73cbd4
I20260620 21:59:04.323975 11781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8cf0f579634c48a3413d224c10c325" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42247 } }
I20260620 21:59:04.324163 11781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:04.324200 11781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8cf0f579634c48a3413d224c10c325, State: Initialized, Role: FOLLOWER
I20260620 21:59:04.324335 11781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8cf0f579634c48a3413d224c10c325" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42247 } }
I20260620 21:59:04.324427 11781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:04.324469 11781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:04.324512 11781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:04.325141 11781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8cf0f579634c48a3413d224c10c325" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42247 } }
I20260620 21:59:04.325278 11781 leader_election.cc:302] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a8cf0f579634c48a3413d224c10c325; no voters:
I20260620 21:59:04.325451 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.322910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.6.241.190:42247)
0620 21:59:04.323156 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.323167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.323201 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:59:04.323928 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.323938 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.324004 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.324167 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:04.324917 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:04.324921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:04.324921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:04.324925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:04.324978 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:04.324990 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:04.325070 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.325141 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.325203 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260620 21:59:04.325500 11781 leader_election.cc:288] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:04.325778 11781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:04.325784 11786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:04.325917 11786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [term 1 LEADER]: Becoming Leader. State: Replica: 8a8cf0f579634c48a3413d224c10c325, State: Running, Role: LEADER
I20260620 21:59:04.326011 11786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8cf0f579634c48a3413d224c10c325" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42247 } }
I20260620 21:59:04.326310 11786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a8cf0f579634c48a3413d224c10c325. Latest consensus state: current_term: 1 leader_uuid: "8a8cf0f579634c48a3413d224c10c325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8cf0f579634c48a3413d224c10c325" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42247 } } }
I20260620 21:59:04.326332 11787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a8cf0f579634c48a3413d224c10c325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8cf0f579634c48a3413d224c10c325" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42247 } } }
I20260620 21:59:04.326398 11786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:04.326424 11787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:04.326583 11795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:04.327191 11795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:04.328836 11795 catalog_manager.cc:1347] Generated new cluster ID: db6046ac0b6b445788a22bc7fd0bd28d
I20260620 21:59:04.328882 11795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:04.341128 11795 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:04.341341 11795 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:04.350838 11795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325: Generated new TSK 0
I20260620 21:59:04.351101 11795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:04.359175 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42247
--builtin_ntp_servers=127.6.241.148:44643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:59:04.444659 11805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:04.444892 11805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:04.444918 11805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:04.446187 11805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:04.446231 11805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:04.446259 11805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:04.447732 11805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:04.448019 11805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:04.448355 11805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:04.454484 11811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:04.454465 11810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:04.454632 11805 server_base.cc:1034] running on GCE node
W20260620 21:59:04.454463 11813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:04.454978 11805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:04.455268 11805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:04.456446 11805 hybrid_clock.cc:648] HybridClock initialized: now 1781992744456424 us; error 43 us; skew 500 ppm
I20260620 21:59:04.457780 11805 webserver.cc:460] Webserver started at http://127.6.241.129:34495/ using document root <none> and password file <none>
I20260620 21:59:04.458105 11805 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:04.458158 11805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:04.458426 11805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:04.459677 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "c42d64d4750343b086423ed1ae778662"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "b88559a6f1f1bfba2623887203f79006"
server_key_iv: "5605480e1354297e0413b5d7d1723383"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.460063 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "c42d64d4750343b086423ed1ae778662"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "b88559a6f1f1bfba2623887203f79006"
server_key_iv: "5605480e1354297e0413b5d7d1723383"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.461442 11805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:04.462196 11819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.462378 11805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:04.462479 11805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "c42d64d4750343b086423ed1ae778662"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "b88559a6f1f1bfba2623887203f79006"
server_key_iv: "5605480e1354297e0413b5d7d1723383"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.462582 11805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:04.466492 11805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:04.466806 11805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:04.466965 11805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:04.467185 11805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:04.467500 11805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:04.467556 11805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.467598 11805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:04.467626 11805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.473572 11805 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:35609
I20260620 21:59:04.473623 11932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:35609 every 8 connection(s)
I20260620 21:59:04.474112 11805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:04.474238 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 11805
I20260620 21:59:04.474344 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:04.474591 7110 external_mini_cluster.cc:1428] Setting key 92af738cdbdb95900c09a25829ddba2c
I20260620 21:59:04.475775 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42247
--builtin_ntp_servers=127.6.241.148:44643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 21:59:04.477908 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.474942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:34621 (local address 127.6.241.190:42247)
0620 21:59:04.475015 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.475018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.475304 (+ 286us) server_negotiation.cc:407] Connection header received
0620 21:59:04.476092 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.476093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.476101 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.476180 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:04.477641 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:04.477643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:04.477643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:04.477649 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:04.477666 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:04.477672 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:04.477733 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.477812 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.477829 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:59:04.478142 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.474975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42247 (local address 127.6.241.129:34621)
0620 21:59:04.475173 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.475208 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:04.475945 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:04.476267 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:04.476272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:04.476284 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:04.477472 (+ 1188us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:04.477472 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:04.477504 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:04.477541 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:04.477733 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:04.477739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:04.477742 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:04.477824 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:59:04.477888 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260620 21:59:04.478653 11933 heartbeater.cc:347] Connected to a master server at 127.6.241.190:42247
I20260620 21:59:04.478745 11933 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:04.478916 11933 heartbeater.cc:511] Master 127.6.241.190:42247 requested a full tablet report, sending...
I20260620 21:59:04.479437 11745 ts_manager.cc:194] Registered new tserver with Master: c42d64d4750343b086423ed1ae778662 (127.6.241.129:35609)
W20260620 21:59:04.559610 11936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:04.559902 11936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:04.559942 11936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:04.561326 11936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:04.561399 11936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:04.561444 11936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:04.562994 11936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:04.563325 11936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:04.563675 11936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:04.569129 11944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:04.569193 11941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:04.569144 11942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:04.569628 11936 server_base.cc:1034] running on GCE node
I20260620 21:59:04.569870 11936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:04.570180 11936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:04.571360 11936 hybrid_clock.cc:648] HybridClock initialized: now 1781992744571315 us; error 59 us; skew 500 ppm
I20260620 21:59:04.575716 11936 webserver.cc:460] Webserver started at http://127.6.241.130:45493/ using document root <none> and password file <none>
I20260620 21:59:04.576036 11936 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:04.576092 11936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:04.576274 11936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:04.577462 11936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "0a3dcb0b40bb409fad3ba2b575a9619b"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "785119fa1848f53ce77f48b8fd43863f"
server_key_iv: "f53980468e4eaa65ae41b7bba27584b3"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.577842 11936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "0a3dcb0b40bb409fad3ba2b575a9619b"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "785119fa1848f53ce77f48b8fd43863f"
server_key_iv: "f53980468e4eaa65ae41b7bba27584b3"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.579363 11936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:04.580245 11950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.580518 11936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:04.580657 11936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "0a3dcb0b40bb409fad3ba2b575a9619b"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "785119fa1848f53ce77f48b8fd43863f"
server_key_iv: "f53980468e4eaa65ae41b7bba27584b3"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.580749 11936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:04.584511 11936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:04.584823 11936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:04.584986 11936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:04.585213 11936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:04.585541 11936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:04.585595 11936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.585636 11936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:04.585662 11936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.592121 11936 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:37959
I20260620 21:59:04.592163 12063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:37959 every 8 connection(s)
I20260620 21:59:04.592551 11936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:04.595916 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.593349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:39525 (local address 127.6.241.190:42247)
0620 21:59:04.593394 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.593396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.593649 (+ 253us) server_negotiation.cc:407] Connection header received
0620 21:59:04.594220 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.594222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.594230 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.594281 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:04.595608 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:04.595609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:04.595609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:04.595615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:04.595634 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:04.595639 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:04.595699 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.595802 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.595819 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 21:59:04.596143 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.593328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42247 (local address 127.6.241.130:39525)
0620 21:59:04.593524 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.593558 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:04.594098 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:04.594364 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:04.594368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:04.594376 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:04.595436 (+ 1060us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:04.595436 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:04.595466 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:04.595495 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:04.595707 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:04.595712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:04.595716 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:04.595813 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 21:59:04.595891 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260620 21:59:04.596674 12064 heartbeater.cc:347] Connected to a master server at 127.6.241.190:42247
I20260620 21:59:04.596755 12064 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:04.596870 12064 heartbeater.cc:511] Master 127.6.241.190:42247 requested a full tablet report, sending...
I20260620 21:59:04.597280 11745 ts_manager.cc:194] Registered new tserver with Master: 0a3dcb0b40bb409fad3ba2b575a9619b (127.6.241.130:37959)
I20260620 21:59:04.601383 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 11936
I20260620 21:59:04.601464 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:04.601680 7110 external_mini_cluster.cc:1428] Setting key 527b33d03262df16cd556292d769ac15
I20260620 21:59:04.602662 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42247
--builtin_ntp_servers=127.6.241.148:44643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:59:04.685667 12067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:04.685902 12067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:04.685927 12067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:04.687341 12067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:04.687392 12067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:04.687422 12067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:04.688877 12067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:04.689163 12067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:04.689497 12067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:04.694751 12073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:04.694759 12072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:04.694766 12075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:04.695052 12067 server_base.cc:1034] running on GCE node
I20260620 21:59:04.695360 12067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:04.695698 12067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:04.696863 12067 hybrid_clock.cc:648] HybridClock initialized: now 1781992744696847 us; error 66 us; skew 500 ppm
I20260620 21:59:04.698186 12067 webserver.cc:460] Webserver started at http://127.6.241.131:34253/ using document root <none> and password file <none>
I20260620 21:59:04.698462 12067 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:04.698531 12067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:04.698733 12067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:04.699965 12067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "f526940612a04024b2d45ed154463671"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "7c324e8b71a0805890ba3df47dbf29f7"
server_key_iv: "066ae00d29136a85182ed13d659843ec"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.700363 12067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "f526940612a04024b2d45ed154463671"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "7c324e8b71a0805890ba3df47dbf29f7"
server_key_iv: "066ae00d29136a85182ed13d659843ec"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.701783 12067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 21:59:04.702499 12081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.702721 12067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 21:59:04.702822 12067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "f526940612a04024b2d45ed154463671"
format_stamp: "Formatted at 2026-06-20 21:59:04 on dist-test-slave-gl2g"
server_key: "7c324e8b71a0805890ba3df47dbf29f7"
server_key_iv: "066ae00d29136a85182ed13d659843ec"
server_key_version: "encryptionkey@0"
I20260620 21:59:04.702909 12067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:04.707245 12067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:04.707540 12067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:04.707691 12067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:04.707917 12067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:04.708245 12067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:04.708299 12067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.708340 12067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:04.708369 12067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:04.714187 12067 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:34149
I20260620 21:59:04.714262 12194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:34149 every 8 connection(s)
I20260620 21:59:04.714694 12067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:04.717612 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 12067
I20260620 21:59:04.717698 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:04.717748 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.715461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:59749 (local address 127.6.241.190:42247)
0620 21:59:04.715525 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.715527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.715758 (+ 231us) server_negotiation.cc:407] Connection header received
0620 21:59:04.716248 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.716249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.716254 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.716321 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:04.717438 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:04.717440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:04.717440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:04.717444 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:04.717459 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:04.717464 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:04.717552 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.717657 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.717676 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:59:04.717876 7110 external_mini_cluster.cc:1428] Setting key 561864a15b8aaa72ba9017de579503dd
I20260620 21:59:04.717969 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.715430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42247 (local address 127.6.241.131:59749)
0620 21:59:04.715630 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.715680 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:04.716155 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:04.716380 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:04.716383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:04.716390 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:04.717284 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:04.717285 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:04.717314 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:04.717344 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:04.717556 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:04.717563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:04.717568 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:59:04.717668 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 21:59:04.717733 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260620 21:59:04.718510 12195 heartbeater.cc:347] Connected to a master server at 127.6.241.190:42247
I20260620 21:59:04.718650 12195 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:04.718808 12195 heartbeater.cc:511] Master 127.6.241.190:42247 requested a full tablet report, sending...
I20260620 21:59:04.719174 11745 ts_manager.cc:194] Registered new tserver with Master: f526940612a04024b2d45ed154463671 (127.6.241.131:34149)
I20260620 21:59:04.730188 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:04.732820 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.731763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.6.241.190:42247)
0620 21:59:04.731831 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.731833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.731985 (+ 152us) server_negotiation.cc:407] Connection header received
0620 21:59:04.732014 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.732015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.732022 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.732069 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:04.732651 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:04.732652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:04.732652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:04.732655 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:04.732666 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:04.732669 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:04.732709 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.732751 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.732763 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:59:04.734687 11745 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:04.736366 11745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:04.740361 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.738879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.6.241.131:34149)
0620 21:59:04.739040 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.739046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.739112 (+ 66us) server_negotiation.cc:407] Connection header received
0620 21:59:04.739157 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.739159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.739168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.739244 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:04.739975 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:04.739983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:04.739984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:04.739988 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:04.740050 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:04.740087 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:04.740167 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.740257 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.740273 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260620 21:59:04.740366 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.738592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:34149 (local address 127.0.0.1:44624)
0620 21:59:04.739060 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.739070 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:04.739119 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:04.739250 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:04.739256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:04.739264 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:04.739861 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:04.739861 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:04.739880 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:04.739906 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:04.740164 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:04.740170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:04.740174 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:04.740249 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 21:59:04.740263 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":100,"threads_started":1}
I20260620 21:59:04.740641 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.738570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:35609 (local address 127.0.0.1:43162)
0620 21:59:04.738970 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.738980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:04.739015 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:04.739249 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:04.739256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:04.739264 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:04.740206 (+ 942us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:04.740206 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:04.740214 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:04.740226 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:04.740483 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:04.740484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:04.740485 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 21:59:04.740561 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:59:04.740575 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"spinlock_wait_cycles":768,"thread_start_us":363,"threads_started":1}
I20260620 21:59:04.740686 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.738729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.6.241.129:35609)
0620 21:59:04.739097 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.739101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.739112 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:59:04.739142 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.739144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.739155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.739244 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:04.740327 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:04.740334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:04.740334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:04.740339 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:04.740370 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:04.740394 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:04.740451 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.740583 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.740602 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":84,"threads_started":1}
I20260620 21:59:04.741212 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.738567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:37959 (local address 127.0.0.1:57312)
0620 21:59:04.738764 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.738807 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:04.738863 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:04.739771 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:04.739772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:04.739776 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:04.740173 (+ 397us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:04.740174 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:04.740181 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:04.740194 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:04.740867 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:04.740868 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:04.740869 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 21:59:04.741142 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 21:59:04.741155 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260620 21:59:04.741271 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.738717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.6.241.130:37959)
0620 21:59:04.739631 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:04.739635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:04.739643 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:04.739663 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:04.739665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:04.739671 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:04.739763 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:04.740580 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:04.740587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:04.740588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:04.740591 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:04.740625 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:04.740648 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:04.740874 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:04.741171 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 21:59:04.741191 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":81,"threads_started":1}
I20260620 21:59:04.741387 11867 tablet_service.cc:1467] Processing CreateTablet for tablet e3c732018403450e815a960270065984 (DEFAULT_TABLE table=test-table [id=84653bfbcbce42bfb62629eabfb91dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:04.741482 12129 tablet_service.cc:1467] Processing CreateTablet for tablet e3c732018403450e815a960270065984 (DEFAULT_TABLE table=test-table [id=84653bfbcbce42bfb62629eabfb91dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:04.741765 11867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3c732018403450e815a960270065984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:04.741832 12129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3c732018403450e815a960270065984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:04.741889 11998 tablet_service.cc:1467] Processing CreateTablet for tablet e3c732018403450e815a960270065984 (DEFAULT_TABLE table=test-table [id=84653bfbcbce42bfb62629eabfb91dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:04.742192 11998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3c732018403450e815a960270065984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:04.744390 12209 tablet_bootstrap.cc:492] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662: Bootstrap starting.
I20260620 21:59:04.745115 12209 tablet_bootstrap.cc:654] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:04.745420 12209 log.cc:824] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:04.745544 12210 tablet_bootstrap.cc:492] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: Bootstrap starting.
I20260620 21:59:04.746106 12210 tablet_bootstrap.cc:654] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:04.746387 12210 log.cc:824] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:04.746757 12211 tablet_bootstrap.cc:492] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: Bootstrap starting.
I20260620 21:59:04.747130 12210 tablet_bootstrap.cc:492] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: No bootstrap required, opened a new log
I20260620 21:59:04.747185 12210 ts_tablet_manager.cc:1397] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:04.747406 12211 tablet_bootstrap.cc:654] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:04.747689 12211 log.cc:824] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:04.748374 12211 tablet_bootstrap.cc:492] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: No bootstrap required, opened a new log
I20260620 21:59:04.748461 12211 ts_tablet_manager.cc:1397] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:04.749483 12210 raft_consensus.cc:348] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } }
I20260620 21:59:04.749639 12210 raft_consensus.cc:374] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:04.749661 12210 raft_consensus.cc:729] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f526940612a04024b2d45ed154463671, State: Initialized, Role: FOLLOWER
I20260620 21:59:04.749805 12210 consensus_queue.cc:260] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } }
I20260620 21:59:04.750135 12210 ts_tablet_manager.cc:1428] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:04.750135 12195 heartbeater.cc:503] Master 127.6.241.190:42247 was elected leader, sending a full tablet report...
I20260620 21:59:04.751147 12209 tablet_bootstrap.cc:492] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662: No bootstrap required, opened a new log
I20260620 21:59:04.751070 12211 raft_consensus.cc:348] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } }
I20260620 21:59:04.751204 12211 raft_consensus.cc:374] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:04.751233 12209 ts_tablet_manager.cc:1397] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 21:59:04.751271 12211 raft_consensus.cc:729] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3dcb0b40bb409fad3ba2b575a9619b, State: Initialized, Role: FOLLOWER
I20260620 21:59:04.751410 12211 consensus_queue.cc:260] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } }
I20260620 21:59:04.751708 12211 ts_tablet_manager.cc:1428] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:04.751814 12064 heartbeater.cc:503] Master 127.6.241.190:42247 was elected leader, sending a full tablet report...
I20260620 21:59:04.753494 12209 raft_consensus.cc:348] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } }
I20260620 21:59:04.753659 12209 raft_consensus.cc:374] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:04.753705 12209 raft_consensus.cc:729] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42d64d4750343b086423ed1ae778662, State: Initialized, Role: FOLLOWER
I20260620 21:59:04.753834 12209 consensus_queue.cc:260] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } }
I20260620 21:59:04.754110 12209 ts_tablet_manager.cc:1428] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:04.754284 11933 heartbeater.cc:503] Master 127.6.241.190:42247 was elected leader, sending a full tablet report...
W20260620 21:59:04.843142 12065 tablet.cc:2369] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:59:04.965289 12196 tablet.cc:2369] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:59:04.974887 11934 tablet.cc:2369] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:05.029932 12215 raft_consensus.cc:482] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:05.030123 12215 raft_consensus.cc:504] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } }
I20260620 21:59:05.030496 12215 leader_election.cc:288] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a3dcb0b40bb409fad3ba2b575a9619b (127.6.241.130:37959), c42d64d4750343b086423ed1ae778662 (127.6.241.129:35609)
I20260620 21:59:05.032034 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.030707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:35609 (local address 127.6.241.131:42451)
0620 21:59:05.030766 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:05.030779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:05.030823 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:05.031097 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:05.031098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:05.031100 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:05.031620 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:05.031620 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:05.031628 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:05.031643 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:05.031880 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:05.031881 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:05.031882 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 21:59:05.031938 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 21:59:05.031952 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 21:59:05.032078 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.030916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:42451 (local address 127.6.241.129:35609)
0620 21:59:05.030977 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.030979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.030988 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:05.031017 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.031018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.031024 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.031086 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:05.031758 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:05.031760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:05.031760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:05.031765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:05.031785 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:05.031790 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:05.031857 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:05.031958 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 21:59:05.031985 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 21:59:05.032181 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.030716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:37959 (local address 127.6.241.131:38441)
0620 21:59:05.030939 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:05.030951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:05.031004 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:05.031160 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:05.031161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:05.031164 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:05.031800 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:05.031800 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:05.031807 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:05.031819 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:05.032001 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:05.032002 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:05.032002 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 21:59:05.032076 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 21:59:05.032089 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":40,"thread_start_us":110,"threads_started":1}
I20260620 21:59:05.032198 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.030858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:38441 (local address 127.6.241.130:37959)
0620 21:59:05.030901 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.030904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.031019 (+ 115us) server_negotiation.cc:407] Connection header received
0620 21:59:05.031078 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.031079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.031084 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.031152 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:05.031906 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:05.031908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:05.031908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:05.031913 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:05.031932 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:05.031937 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:05.031994 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:05.032093 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 21:59:05.032108 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 21:59:05.032407 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c732018403450e815a960270065984" candidate_uuid: "f526940612a04024b2d45ed154463671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42d64d4750343b086423ed1ae778662" is_pre_election: true
I20260620 21:59:05.032454 12018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c732018403450e815a960270065984" candidate_uuid: "f526940612a04024b2d45ed154463671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" is_pre_election: true
I20260620 21:59:05.032552 11887 raft_consensus.cc:2454] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f526940612a04024b2d45ed154463671 in term 0.
I20260620 21:59:05.032617 12018 raft_consensus.cc:2454] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f526940612a04024b2d45ed154463671 in term 0.
I20260620 21:59:05.032766 12083 leader_election.cc:302] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c42d64d4750343b086423ed1ae778662, f526940612a04024b2d45ed154463671; no voters:
I20260620 21:59:05.032892 12215 raft_consensus.cc:2781] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:05.032967 12215 raft_consensus.cc:482] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:05.032984 12215 raft_consensus.cc:3037] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:05.033725 12215 raft_consensus.cc:504] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } }
I20260620 21:59:05.033861 12215 leader_election.cc:288] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [CANDIDATE]: Term 1 election: Requested vote from peers 0a3dcb0b40bb409fad3ba2b575a9619b (127.6.241.130:37959), c42d64d4750343b086423ed1ae778662 (127.6.241.129:35609)
I20260620 21:59:05.033987 12018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c732018403450e815a960270065984" candidate_uuid: "f526940612a04024b2d45ed154463671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b"
I20260620 21:59:05.034066 12018 raft_consensus.cc:3037] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:05.034085 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c732018403450e815a960270065984" candidate_uuid: "f526940612a04024b2d45ed154463671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42d64d4750343b086423ed1ae778662"
I20260620 21:59:05.034158 11887 raft_consensus.cc:3037] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:05.034868 12018 raft_consensus.cc:2454] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f526940612a04024b2d45ed154463671 in term 1.
I20260620 21:59:05.034986 11887 raft_consensus.cc:2454] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f526940612a04024b2d45ed154463671 in term 1.
I20260620 21:59:05.035032 12085 leader_election.cc:302] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3dcb0b40bb409fad3ba2b575a9619b, f526940612a04024b2d45ed154463671; no voters:
I20260620 21:59:05.035146 12215 raft_consensus.cc:2781] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:05.035320 12215 raft_consensus.cc:686] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 1 LEADER]: Becoming Leader. State: Replica: f526940612a04024b2d45ed154463671, State: Running, Role: LEADER
I20260620 21:59:05.035432 12215 consensus_queue.cc:237] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } }
I20260620 21:59:05.036355 11745 catalog_manager.cc:5495] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 reported cstate change: term changed from 0 to 1, leader changed from <none> to f526940612a04024b2d45ed154463671 (127.6.241.131). New cstate: current_term: 1 leader_uuid: "f526940612a04024b2d45ed154463671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f526940612a04024b2d45ed154463671" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 } health_report { overall_health: UNKNOWN } } }
I20260620 21:59:05.099714 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.098867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.6.241.131:34149)
0620 21:59:05.098932 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.098934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.098940 (+ 6us) server_negotiation.cc:407] Connection header received
0620 21:59:05.098974 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.098975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.098987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.099030 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:05.099529 (+ 499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:05.099530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:05.099531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:05.099535 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:05.099548 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:05.099552 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:05.099599 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:05.099644 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 21:59:05.099656 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:59:05.101790 12018 raft_consensus.cc:1261] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [term 1 FOLLOWER]: Refusing update from remote peer f526940612a04024b2d45ed154463671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:05.101789 11887 raft_consensus.cc:1261] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [term 1 FOLLOWER]: Refusing update from remote peer f526940612a04024b2d45ed154463671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:05.102172 12219 consensus_queue.cc:1035] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42d64d4750343b086423ed1ae778662" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:05.102306 12215 consensus_queue.cc:1035] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a3dcb0b40bb409fad3ba2b575a9619b" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:05.104987 12223 mvcc.cc:205] Tried to move back new op lower bound from 7299042283933556736 to 7299042283665862656. Current Snapshot: MvccSnapshot[applied={T|T < 7299042283933556736}]
I20260620 21:59:05.105020 12224 mvcc.cc:205] Tried to move back new op lower bound from 7299042283933556736 to 7299042283665862656. Current Snapshot: MvccSnapshot[applied={T|T < 7299042283933556736}]
I20260620 21:59:05.107161 11745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:05.107252 11745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:05.108019 11745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a8cf0f579634c48a3413d224c10c325: Sending DeleteTablet for 3 replicas of tablet e3c732018403450e815a960270065984
I20260620 21:59:05.108361 11867 tablet_service.cc:1514] Processing DeleteTablet for tablet e3c732018403450e815a960270065984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:05 UTC) from {username='slave'} at 127.0.0.1:43162
I20260620 21:59:05.108358 11998 tablet_service.cc:1514] Processing DeleteTablet for tablet e3c732018403450e815a960270065984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:05 UTC) from {username='slave'} at 127.0.0.1:57312
I20260620 21:59:05.108379 12129 tablet_service.cc:1514] Processing DeleteTablet for tablet e3c732018403450e815a960270065984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:05 UTC) from {username='slave'} at 127.0.0.1:44624
I20260620 21:59:05.108623 12232 tablet_replica.cc:330] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: stopping tablet replica
I20260620 21:59:05.108723 12233 tablet_replica.cc:330] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662: stopping tablet replica
I20260620 21:59:05.108826 12233 raft_consensus.cc:2229] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:05.108892 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 11805
I20260620 21:59:05.108948 12233 raft_consensus.cc:2258] T e3c732018403450e815a960270065984 P c42d64d4750343b086423ed1ae778662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:05.109105 12234 tablet_replica.cc:330] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: stopping tablet replica
I20260620 21:59:05.109189 12234 raft_consensus.cc:2229] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:05.109274 12234 raft_consensus.cc:2258] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:05.109854 12234 ts_tablet_manager.cc:1905] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:05.111522 12232 raft_consensus.cc:2229] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:05.111665 12232 raft_consensus.cc:2258] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:05.112146 12232 ts_tablet_manager.cc:1905] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:05.114173 12232 ts_tablet_manager.cc:1918] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:05.114259 12232 log.cc:1196] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/e3c732018403450e815a960270065984
I20260620 21:59:05.114521 12232 ts_tablet_manager.cc:1939] T e3c732018403450e815a960270065984 P 0a3dcb0b40bb409fad3ba2b575a9619b: Deleting consensus metadata
I20260620 21:59:05.115042 11732 catalog_manager.cc:4841] TS 0a3dcb0b40bb409fad3ba2b575a9619b (127.6.241.130:37959): tablet e3c732018403450e815a960270065984 (table test-table [id=84653bfbcbce42bfb62629eabfb91dc6]) successfully deleted
W20260620 21:59:05.116045 11730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:35609 (error 108)
I20260620 21:59:05.116258 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 11936
I20260620 21:59:05.116303 12234 ts_tablet_manager.cc:1918] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:05.116380 12234 log.cc:1196] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/e3c732018403450e815a960270065984
I20260620 21:59:05.116652 12234 ts_tablet_manager.cc:1939] T e3c732018403450e815a960270065984 P f526940612a04024b2d45ed154463671: Deleting consensus metadata
I20260620 21:59:05.117591 11731 catalog_manager.cc:4841] TS f526940612a04024b2d45ed154463671 (127.6.241.131:34149): tablet e3c732018403450e815a960270065984 (table test-table [id=84653bfbcbce42bfb62629eabfb91dc6]) successfully deleted
I20260620 21:59:05.119273 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.119111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:35609 (local address 127.0.0.1:43174)
0620 21:59:05.119179 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:05.119231 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:59:05.119411 11730 catalog_manager.cc:4584] TS c42d64d4750343b086423ed1ae778662 (127.6.241.129:35609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3c732018403450e815a960270065984: Network error: Client connection negotiation failed: client connection to 127.6.241.129:35609: connect: Connection refused (error 111)
I20260620 21:59:05.122192 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 12067
I20260620 21:59:05.127405 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 11714
2026-06-20T21:59:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (946 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 21:59:05.147243 7110 mini_oidc.cc:152] Starting JWKS server
I20260620 21:59:05.148329 7110 webserver.cc:460] Webserver started at https://127.0.0.1:36301/ using document root <none> and password file <none>
I20260620 21:59:05.148430 7110 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:59:05.148605 7110 webserver.cc:460] Webserver started at http://127.0.0.1:33993/ using document root <none> and password file <none>
2026-06-20T21:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:05Z Disabled control of system clock
I20260620 21:59:05.156329 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34989
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:34989
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36301/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/testchainca.cert with env {}
W20260620 21:59:05.237227 12241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:05.237471 12241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:05.237496 12241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:05.238734 12241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:05.238777 12241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:05.238790 12241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:05.238803 12241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:05.238816 12241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:59:05.238828 12241 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:59:05.238840 12241 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36301/jwks.json
I20260620 21:59:05.240492 12241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:34989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36301/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:05.240710 12241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:05.240878 12241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:05.245992 12246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:05.245960 12247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:05.246085 12241 server_base.cc:1034] running on GCE node
W20260620 21:59:05.245970 12249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:05.246465 12241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:05.246826 12241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:05.248044 12241 hybrid_clock.cc:648] HybridClock initialized: now 1781992745248015 us; error 59 us; skew 500 ppm
I20260620 21:59:05.249472 12241 webserver.cc:460] Webserver started at http://127.6.241.190:38139/ using document root <none> and password file <none>
I20260620 21:59:05.249748 12241 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:05.249825 12241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:05.250006 12241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:05.251255 12241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "07b6078f6e36449695a9e027cf59ed3b"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "df69678e812576ac9cb5d76e4c3006fb"
server_key_iv: "3831c00bee4f1bd079e129f46fa23a26"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.251643 12241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "07b6078f6e36449695a9e027cf59ed3b"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "df69678e812576ac9cb5d76e4c3006fb"
server_key_iv: "3831c00bee4f1bd079e129f46fa23a26"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.253049 12241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:05.253901 12255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:05.254158 12241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:05.254285 12241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "07b6078f6e36449695a9e027cf59ed3b"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "df69678e812576ac9cb5d76e4c3006fb"
server_key_iv: "3831c00bee4f1bd079e129f46fa23a26"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.254387 12241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:05.269569 12241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:05.269940 12241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:05.270116 12241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:05.274345 12241 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:34989
I20260620 21:59:05.274384 12307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:34989 every 8 connection(s)
I20260620 21:59:05.274808 12241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:05.275823 12308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:05.278208 12308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b: Bootstrap starting.
I20260620 21:59:05.278914 12308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:05.279228 12308 log.cc:824] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:05.279982 12308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b: No bootstrap required, opened a new log
I20260620 21:59:05.280774 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 12241
I20260620 21:59:05.280858 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:05.281049 7110 external_mini_cluster.cc:1428] Setting key f5434da4ab0f5c86b69ffd44661a2cd1
I20260620 21:59:05.282217 12308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b6078f6e36449695a9e027cf59ed3b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34989 } }
I20260620 21:59:05.282395 12308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:05.282450 12308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b6078f6e36449695a9e027cf59ed3b, State: Initialized, Role: FOLLOWER
I20260620 21:59:05.282578 12308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b6078f6e36449695a9e027cf59ed3b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34989 } }
I20260620 21:59:05.282729 12308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:05.282763 12308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:05.282810 12308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:05.283581 12308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b6078f6e36449695a9e027cf59ed3b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34989 } }
I20260620 21:59:05.283690 12308 leader_election.cc:302] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07b6078f6e36449695a9e027cf59ed3b; no voters:
I20260620 21:59:05.283843 12308 leader_election.cc:288] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:05.283892 12313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:05.284036 12313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [term 1 LEADER]: Becoming Leader. State: Replica: 07b6078f6e36449695a9e027cf59ed3b, State: Running, Role: LEADER
I20260620 21:59:05.284101 12308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:05.284122 12313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b6078f6e36449695a9e027cf59ed3b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34989 } }
I20260620 21:59:05.284458 12313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07b6078f6e36449695a9e027cf59ed3b. Latest consensus state: current_term: 1 leader_uuid: "07b6078f6e36449695a9e027cf59ed3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b6078f6e36449695a9e027cf59ed3b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34989 } } }
I20260620 21:59:05.284569 12313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:05.285569 12319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:05.284467 12314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07b6078f6e36449695a9e027cf59ed3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b6078f6e36449695a9e027cf59ed3b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34989 } } }
I20260620 21:59:05.285866 12314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:05.286193 12319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:05.288061 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.281616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.6.241.190:34989)
0620 21:59:05.281852 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.281862 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.281900 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:59:05.282178 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.282186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.282266 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.282433 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:05.282728 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.284532 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.285591 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.286210 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.286352 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.287406 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:05.287410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:05.287410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:05.287598 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:05.287631 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:05.287642 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:05.287728 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:05.287803 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 21:59:05.287882 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260620 21:59:05.289027 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:05.291052 12319 catalog_manager.cc:1347] Generated new cluster ID: 0fab9cbdba3a46aa81369ca2bad58b82
I20260620 21:59:05.291100 12319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:05.309502 12319 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:05.310124 12319 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:05.321249 12319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07b6078f6e36449695a9e027cf59ed3b: Generated new TSK 0
I20260620 21:59:05.321519 12319 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 21:59:05.343400 12338 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 21:59:05.343526 12338 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 21:59:05.344007 12337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:05.328508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:34989 (local address 127.0.0.1:53246)
0620 21:59:05.328705 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:05.328722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:05.328817 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:05.329042 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:05.329043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:05.329045 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:59:05.329119 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:05.329121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.329792 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.329793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:05.330428 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:05.330430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.330867 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.330868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:05.330933 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.330961 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:59:05.343680 (+ 12719us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 21:59:05.343771 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.241.190:34989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260620 21:59:05.344450 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.328621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.6.241.190:34989)
0620 21:59:05.328673 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.328676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.328791 (+ 115us) server_negotiation.cc:407] Connection header received
0620 21:59:05.328928 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.328929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.328943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.329030 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:59:05.329209 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.329690 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.330524 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.330781 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.330865 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.331099 (+ 234us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:59:05.343519 (+ 12420us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 21:59:05.344372 (+ 853us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 21:59:05.344499 12312 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.241.190:34989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 21:59:05.344568 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 12241
2026-06-20T21:59:05Z chronyd exiting
I20260620 21:59:05.380633 7110 test_util.cc:175] -----------------------------------------------
I20260620 21:59:05.380746 7110 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992718478901-7110-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 21:59:05.381558 7110 mini_oidc.cc:152] Starting JWKS server
I20260620 21:59:05.382269 7110 webserver.cc:460] Webserver started at https://127.0.0.1:46071/ using document root <none> and password file <none>
I20260620 21:59:05.382339 7110 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:59:05.382525 7110 webserver.cc:460] Webserver started at http://127.0.0.1:34261/ using document root <none> and password file <none>
2026-06-20T21:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:05Z Disabled control of system clock
I20260620 21:59:05.390396 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40827
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:42817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:40827
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46071/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/testchainca.cert with env {}
W20260620 21:59:05.472482 12344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:05.472718 12344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:05.472743 12344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:05.473986 12344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:05.474030 12344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:05.474042 12344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:05.474056 12344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:05.474067 12344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:59:05.474123 12344 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:59:05.474143 12344 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46071/jwks.json
I20260620 21:59:05.475687 12344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:42817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:40827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46071/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:05.475926 12344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:05.476102 12344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:05.481163 12350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:05.481161 12352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:05.481163 12349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:05.481380 12344 server_base.cc:1034] running on GCE node
I20260620 21:59:05.481721 12344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:05.482097 12344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:05.483275 12344 hybrid_clock.cc:648] HybridClock initialized: now 1781992745483250 us; error 37 us; skew 500 ppm
I20260620 21:59:05.484663 12344 webserver.cc:460] Webserver started at http://127.6.241.190:33595/ using document root <none> and password file <none>
I20260620 21:59:05.484931 12344 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:05.485001 12344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:05.485172 12344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:05.486300 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "3e6ec40a1de74703bb21d669256c822c"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "42336f59aeb37ec32ad09618301e0ce2"
server_key_iv: "14c71e61ac2581fa7bf4417a02ad6a20"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.486752 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "3e6ec40a1de74703bb21d669256c822c"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "42336f59aeb37ec32ad09618301e0ce2"
server_key_iv: "14c71e61ac2581fa7bf4417a02ad6a20"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.488169 12344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:05.489053 12358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:05.489286 12344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:05.489393 12344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "3e6ec40a1de74703bb21d669256c822c"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "42336f59aeb37ec32ad09618301e0ce2"
server_key_iv: "14c71e61ac2581fa7bf4417a02ad6a20"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.489487 12344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:05.503856 12344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:05.504220 12344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:05.504379 12344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:05.508553 12344 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:40827
I20260620 21:59:05.508596 12410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:40827 every 8 connection(s)
I20260620 21:59:05.508970 12344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:05.509931 12411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:05.512347 12411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c: Bootstrap starting.
I20260620 21:59:05.512959 12411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:05.513257 12411 log.cc:824] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:05.513963 12411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c: No bootstrap required, opened a new log
I20260620 21:59:05.514827 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 12344
I20260620 21:59:05.514912 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:05.515128 7110 external_mini_cluster.cc:1428] Setting key 68194573849954e900fabc321a3426c8
I20260620 21:59:05.516176 12411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ec40a1de74703bb21d669256c822c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40827 } }
I20260620 21:59:05.516355 12411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:05.516393 12411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6ec40a1de74703bb21d669256c822c, State: Initialized, Role: FOLLOWER
I20260620 21:59:05.516531 12411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ec40a1de74703bb21d669256c822c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40827 } }
I20260620 21:59:05.516619 12411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:05.516657 12411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:05.516708 12411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:05.517503 12411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ec40a1de74703bb21d669256c822c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40827 } }
I20260620 21:59:05.517617 12411 leader_election.cc:302] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e6ec40a1de74703bb21d669256c822c; no voters:
I20260620 21:59:05.517763 12411 leader_election.cc:288] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:05.517817 12416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:05.517958 12416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [term 1 LEADER]: Becoming Leader. State: Replica: 3e6ec40a1de74703bb21d669256c822c, State: Running, Role: LEADER
I20260620 21:59:05.517997 12411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:05.518054 12416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ec40a1de74703bb21d669256c822c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40827 } }
I20260620 21:59:05.518473 12417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e6ec40a1de74703bb21d669256c822c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ec40a1de74703bb21d669256c822c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40827 } } }
I20260620 21:59:05.518561 12417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:05.518790 12418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e6ec40a1de74703bb21d669256c822c. Latest consensus state: current_term: 1 leader_uuid: "3e6ec40a1de74703bb21d669256c822c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ec40a1de74703bb21d669256c822c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40827 } } }
I20260620 21:59:05.518898 12418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:05.519821 12432 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:05.519881 12432 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:05.519924 12425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:05.520066 12425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:05.521627 12425 catalog_manager.cc:1347] Generated new cluster ID: 4970b657f5404324bc6e4bb3496e8f03
I20260620 21:59:05.521682 12425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:05.522074 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.515638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.6.241.190:40827)
0620 21:59:05.515838 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.515845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.515871 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:59:05.516136 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.516142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.516204 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.516315 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:05.516604 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.518372 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.519726 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.520309 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.520429 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.521340 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:05.521344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:05.521345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:05.521559 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:05.521606 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:05.521621 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:05.521770 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:05.521807 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 21:59:05.521894 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260620 21:59:05.523049 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:05.548738 12425 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:05.549329 12425 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:05.553887 12425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e6ec40a1de74703bb21d669256c822c: Generated new TSK 0
I20260620 21:59:05.554117 12425 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 21:59:05.581737 12441 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 21:59:05.581847 12441 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 21:59:05.582201 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.567494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.6.241.190:40827)
0620 21:59:05.567547 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.567550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.567647 (+ 97us) server_negotiation.cc:407] Connection header received
0620 21:59:05.567698 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.567699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.567713 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.567804 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:59:05.567932 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.568703 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.569521 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.569836 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.569911 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.570131 (+ 220us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:59:05.581948 (+ 11817us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 21:59:05.582123 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 21:59:05.582264 12415 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 21:59:05.582294 12440 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:05.567408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40827 (local address 127.0.0.1:45804)
0620 21:59:05.567561 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:05.567572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:05.567638 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:05.567808 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:05.567808 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:05.567810 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:59:05.567860 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:05.567862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.568813 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.568813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:05.569421 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:05.569423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.569943 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.569944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:05.569996 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.570002 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:59:05.582084 (+ 12082us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 21:59:05.582189 (+ 105us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.241.190:40827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260620 21:59:05.582646 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 12344
2026-06-20T21:59:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 21:59:05.620582 7110 mini_oidc.cc:152] Starting JWKS server
I20260620 21:59:05.621325 7110 webserver.cc:460] Webserver started at https://127.0.0.1:46319/ using document root <none> and password file <none>
I20260620 21:59:05.621373 7110 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:59:05.621553 7110 webserver.cc:460] Webserver started at http://127.0.0.1:34817/ using document root <none> and password file <none>
2026-06-20T21:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:05Z Disabled control of system clock
I20260620 21:59:05.629529 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41785
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:41785
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46319/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 21:59:05.711762 12447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:05.711997 12447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:05.712019 12447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:05.713178 12447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:05.713217 12447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:05.713230 12447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:05.713243 12447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:05.713255 12447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:59:05.713267 12447 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:59:05.713279 12447 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46319/jwks.json
I20260620 21:59:05.714751 12447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:41785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46319/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:05.715070 12447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:05.715327 12447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:05.720252 12453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:05.720233 12455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:05.720364 12447 server_base.cc:1034] running on GCE node
W20260620 21:59:05.720233 12452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:05.720674 12447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:05.721064 12447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:05.722276 12447 hybrid_clock.cc:648] HybridClock initialized: now 1781992745722251 us; error 38 us; skew 500 ppm
I20260620 21:59:05.723701 12447 webserver.cc:460] Webserver started at http://127.6.241.190:33257/ using document root <none> and password file <none>
I20260620 21:59:05.723973 12447 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:05.724046 12447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:05.724224 12447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:05.725339 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "2d68373762a143e3afdaf55ce6a0eb87"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "da20b083ad64e8214c271f652b965815"
server_key_iv: "e4f0eeac2d3857828c2a5df8e88c668d"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.725729 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "2d68373762a143e3afdaf55ce6a0eb87"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "da20b083ad64e8214c271f652b965815"
server_key_iv: "e4f0eeac2d3857828c2a5df8e88c668d"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.727182 12447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 21:59:05.727912 12461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:05.728101 12447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 21:59:05.728202 12447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "2d68373762a143e3afdaf55ce6a0eb87"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "da20b083ad64e8214c271f652b965815"
server_key_iv: "e4f0eeac2d3857828c2a5df8e88c668d"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.728287 12447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:05.741374 12447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:05.741706 12447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:05.741860 12447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:05.745791 12447 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:41785
I20260620 21:59:05.745826 12513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:41785 every 8 connection(s)
I20260620 21:59:05.746196 12447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:05.746922 12514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:05.749708 12514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87: Bootstrap starting.
I20260620 21:59:05.750317 12514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:05.750653 12514 log.cc:824] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:05.751348 12514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87: No bootstrap required, opened a new log
I20260620 21:59:05.752964 12514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d68373762a143e3afdaf55ce6a0eb87" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41785 } }
I20260620 21:59:05.753114 12514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:05.753137 12514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d68373762a143e3afdaf55ce6a0eb87, State: Initialized, Role: FOLLOWER
I20260620 21:59:05.753337 12514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d68373762a143e3afdaf55ce6a0eb87" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41785 } }
I20260620 21:59:05.753419 12514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:05.753468 12514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:05.753517 12514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:05.754079 12514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d68373762a143e3afdaf55ce6a0eb87" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41785 } }
I20260620 21:59:05.754225 12514 leader_election.cc:302] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d68373762a143e3afdaf55ce6a0eb87; no voters:
I20260620 21:59:05.754302 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 12447
I20260620 21:59:05.754410 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:05.754408 12514 leader_election.cc:288] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:05.754446 12517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:05.754637 7110 external_mini_cluster.cc:1428] Setting key f00a9aa9874ec20b660d354f01bc723f
I20260620 21:59:05.754635 12517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [term 1 LEADER]: Becoming Leader. State: Replica: 2d68373762a143e3afdaf55ce6a0eb87, State: Running, Role: LEADER
I20260620 21:59:05.754698 12514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:05.754741 12517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d68373762a143e3afdaf55ce6a0eb87" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41785 } }
I20260620 21:59:05.755048 12519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d68373762a143e3afdaf55ce6a0eb87. Latest consensus state: current_term: 1 leader_uuid: "2d68373762a143e3afdaf55ce6a0eb87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d68373762a143e3afdaf55ce6a0eb87" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41785 } } }
I20260620 21:59:05.755205 12519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:05.755249 12518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d68373762a143e3afdaf55ce6a0eb87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d68373762a143e3afdaf55ce6a0eb87" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41785 } } }
I20260620 21:59:05.755314 12518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:05.755522 12528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:05.756105 12528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:05.758926 12528 catalog_manager.cc:1347] Generated new cluster ID: 0b9ff6ec8c984ce690a3fc1cf336352e
I20260620 21:59:05.758985 12528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:05.760819 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.755259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.6.241.190:41785)
0620 21:59:05.755724 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.755733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.755772 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:59:05.755927 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.755936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.756019 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.756173 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:05.756498 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.757633 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.758760 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.759290 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.759426 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.760147 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:05.760151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:05.760151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:05.760334 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:05.760368 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:05.760379 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:05.760456 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:05.760532 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 21:59:05.760629 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":84,"threads_started":1}
I20260620 21:59:05.761659 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:05.764629 12528 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:05.765314 12528 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:05.771102 12528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d68373762a143e3afdaf55ce6a0eb87: Generated new TSK 0
I20260620 21:59:05.771299 12528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:05.780905 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.777237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.6.241.190:41785)
0620 21:59:05.777304 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.777307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.777417 (+ 110us) server_negotiation.cc:407] Connection header received
0620 21:59:05.777508 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.777509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.777522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.777636 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:59:05.777792 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.778322 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.779259 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.779480 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.779565 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.779761 (+ 196us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:59:05.780651 (+ 890us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 21:59:05.780816 (+ 165us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48652: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 21:59:05.780987 12526 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48652: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 21:59:05.781245 12543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:05.777153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41785 (local address 127.0.0.1:48652)
0620 21:59:05.777340 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:05.777350 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:05.777437 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:05.777637 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:05.777638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:05.777641 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:59:05.777704 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:05.777706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.778429 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.778430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:05.779160 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:05.779162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.779562 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.779563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:05.779622 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.779628 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:59:05.780762 (+ 1134us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 21:59:05.780861 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.241.190:41785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
I20260620 21:59:05.781507 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 12447
E20260620 21:59:05.794090 12544 webserver.cc:533] Webserver: sslize failed:
E20260620 21:59:05.794191 12544 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T21:59:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (199 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T21:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:05Z Disabled control of system clock
I20260620 21:59:05.828240 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33397
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:33023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:33397
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/testchainca.cert with env {}
W20260620 21:59:05.909951 12548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:05.910198 12548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:05.910225 12548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:05.911428 12548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:05.911468 12548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:05.911480 12548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:05.911494 12548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:05.911505 12548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:59:05.911518 12548 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:59:05.911530 12548 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 21:59:05.912994 12548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:33023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:33397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:05.913206 12548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:05.913367 12548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:05.918659 12556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:05.918617 12553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:05.918630 12554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:05.918690 12548 server_base.cc:1034] running on GCE node
I20260620 21:59:05.919075 12548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:05.919399 12548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:05.920555 12548 hybrid_clock.cc:648] HybridClock initialized: now 1781992745920540 us; error 27 us; skew 500 ppm
I20260620 21:59:05.921928 12548 webserver.cc:460] Webserver started at http://127.6.241.190:42723/ using document root <none> and password file <none>
I20260620 21:59:05.922205 12548 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:05.922257 12548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:05.922425 12548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:05.923625 12548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "acb7180030bb4775b2058e093a382295"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "5ba0c84fc2466f4732751934aa5ab17d"
server_key_iv: "6c6894326851be8bb8cf723561959b57"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.924005 12548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "acb7180030bb4775b2058e093a382295"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "5ba0c84fc2466f4732751934aa5ab17d"
server_key_iv: "6c6894326851be8bb8cf723561959b57"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.925369 12548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:05.926136 12562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:05.926316 12548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:05.926424 12548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "acb7180030bb4775b2058e093a382295"
format_stamp: "Formatted at 2026-06-20 21:59:05 on dist-test-slave-gl2g"
server_key: "5ba0c84fc2466f4732751934aa5ab17d"
server_key_iv: "6c6894326851be8bb8cf723561959b57"
server_key_version: "encryptionkey@0"
I20260620 21:59:05.926502 12548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:05.957304 12548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:05.957666 12548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:05.957827 12548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:05.961935 12548 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:33397
I20260620 21:59:05.961982 12614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:33397 every 8 connection(s)
I20260620 21:59:05.962374 12548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:05.963438 12615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:05.963563 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 12548
I20260620 21:59:05.963665 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:05.963872 7110 external_mini_cluster.cc:1428] Setting key 718ae265e86c456d185f331e80709b57
I20260620 21:59:05.966830 12615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295: Bootstrap starting.
I20260620 21:59:05.967561 12615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:05.967885 12615 log.cc:824] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:05.968577 12615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295: No bootstrap required, opened a new log
I20260620 21:59:05.969743 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.964528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.6.241.190:33397)
0620 21:59:05.964735 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:05.964742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:05.964768 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:59:05.965234 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:05.965242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:05.965307 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:05.965430 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:05.965656 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.966523 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.967678 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:05.968167 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:05.968335 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:05.969101 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:05.969105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:05.969105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:05.969282 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:05.969316 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:05.969327 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:05.969405 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:05.969488 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 21:59:05.969572 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260620 21:59:05.970755 12615 raft_consensus.cc:348] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb7180030bb4775b2058e093a382295" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33397 } }
I20260620 21:59:05.970908 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:05.970932 12615 raft_consensus.cc:374] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:05.970969 12615 raft_consensus.cc:729] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb7180030bb4775b2058e093a382295, State: Initialized, Role: FOLLOWER
I20260620 21:59:05.971092 12615 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb7180030bb4775b2058e093a382295" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33397 } }
I20260620 21:59:05.971166 12615 raft_consensus.cc:388] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:05.971205 12615 raft_consensus.cc:482] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:05.971254 12615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:05.971946 12615 raft_consensus.cc:504] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb7180030bb4775b2058e093a382295" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33397 } }
I20260620 21:59:05.972056 12615 leader_election.cc:302] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb7180030bb4775b2058e093a382295; no voters:
I20260620 21:59:05.972216 12615 leader_election.cc:288] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:05.972287 12620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:05.972465 12620 raft_consensus.cc:686] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [term 1 LEADER]: Becoming Leader. State: Replica: acb7180030bb4775b2058e093a382295, State: Running, Role: LEADER
I20260620 21:59:05.972494 12615 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:05.972558 12620 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb7180030bb4775b2058e093a382295" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33397 } }
I20260620 21:59:05.972908 12621 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acb7180030bb4775b2058e093a382295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb7180030bb4775b2058e093a382295" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33397 } } }
I20260620 21:59:05.973058 12621 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:05.972940 12622 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb7180030bb4775b2058e093a382295. Latest consensus state: current_term: 1 leader_uuid: "acb7180030bb4775b2058e093a382295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb7180030bb4775b2058e093a382295" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33397 } } }
I20260620 21:59:05.973225 12622 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:05.973546 12629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:05.974030 12629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:05.975700 12629 catalog_manager.cc:1347] Generated new cluster ID: b7902151b45f466280be7a3e193f542b
I20260620 21:59:05.975776 12629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:05.989070 12629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:05.989615 12629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:06.000881 12629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acb7180030bb4775b2058e093a382295: Generated new TSK 0
I20260620 21:59:06.001138 12629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:06.451465 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:06.011869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.6.241.190:33397)
0620 21:59:06.011933 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:06.011935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:06.012046 (+ 111us) server_negotiation.cc:407] Connection header received
0620 21:59:06.012147 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:06.012148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:06.012160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:06.012241 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:59:06.012387 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:06.012886 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:06.013697 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:06.013899 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:06.013975 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:06.014166 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:59:06.451225 (+437059us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 21:59:06.451400 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 21:59:06.451589 12618 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 21:59:06.451591 12644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:06.011791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33397 (local address 127.0.0.1:60746)
0620 21:59:06.011965 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:06.011976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:06.012066 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:06.012240 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:06.012241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:06.012243 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:59:06.012305 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:06.012307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:06.012989 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:06.012990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:06.013604 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:06.013606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:06.013979 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:06.013980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:06.014041 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:06.014046 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:59:06.451379 (+437333us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 21:59:06.451501 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.241.190:33397: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260620 21:59:06.451982 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 12548
2026-06-20T21:59:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (649 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:06 dist-test-slave-gl2g krb5kdc[12656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:06 dist-test-slave-gl2g krb5kdc[12656](info): set up 2 sockets
Jun 20 21:59:06 dist-test-slave-gl2g krb5kdc[12656](info): commencing operation
krb5kdc: starting...
W20260620 21:59:07.703192 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:59:07 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:59:07.823720 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42889
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:42889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:07.906136 12669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:07.906417 12669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:07.906456 12669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:07.907722 12669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:07.907807 12669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:07.907831 12669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:07.907850 12669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:07.907881 12669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:07.909416 12669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:42889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:07.909715 12669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:07.909956 12669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:07.914914 12675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:07.914974 12674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:07.914914 12677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:07.915154 12669 server_base.cc:1034] running on GCE node
I20260620 21:59:07.915405 12669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:07.915766 12669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:07.916976 12669 hybrid_clock.cc:648] HybridClock initialized: now 1781992747916950 us; error 52 us; skew 500 ppm
Jun 20 21:59:07 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992747, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:07.919904 12669 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:59:07.920528 12669 webserver.cc:460] Webserver started at http://127.6.241.190:42159/ using document root <none> and password file <none>
I20260620 21:59:07.920831 12669 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:07.920889 12669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:07.920995 12669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:07.922132 12669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "fa6bff06caf3474692a4aebb2be266c3"
format_stamp: "Formatted at 2026-06-20 21:59:07 on dist-test-slave-gl2g"
server_key: "9039594d2d4a68004fc8e45e9ec7f8a7"
server_key_iv: "b4c0cd50b9f561b387fef3634696f643"
server_key_version: "encryptionkey@0"
I20260620 21:59:07.922462 12669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "fa6bff06caf3474692a4aebb2be266c3"
format_stamp: "Formatted at 2026-06-20 21:59:07 on dist-test-slave-gl2g"
server_key: "9039594d2d4a68004fc8e45e9ec7f8a7"
server_key_iv: "b4c0cd50b9f561b387fef3634696f643"
server_key_version: "encryptionkey@0"
I20260620 21:59:07.924077 12669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:07.925154 12684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:07.925348 12669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:07.925465 12669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "fa6bff06caf3474692a4aebb2be266c3"
format_stamp: "Formatted at 2026-06-20 21:59:07 on dist-test-slave-gl2g"
server_key: "9039594d2d4a68004fc8e45e9ec7f8a7"
server_key_iv: "b4c0cd50b9f561b387fef3634696f643"
server_key_version: "encryptionkey@0"
I20260620 21:59:07.925578 12669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:07.935931 12669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:07.937980 12669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:07.938151 12669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:07.942271 12669 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:42889
I20260620 21:59:07.942330 12736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:42889 every 8 connection(s)
I20260620 21:59:07.942787 12669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:07.943778 12737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:07.946125 12737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3: Bootstrap starting.
I20260620 21:59:07.946784 12737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:07.947077 12737 log.cc:824] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:07.947916 12737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3: No bootstrap required, opened a new log
I20260620 21:59:07.948596 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 12669
I20260620 21:59:07.948709 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:07.948895 7110 external_mini_cluster.cc:1428] Setting key ba1373670760422a65e2ce74b4edd28d
I20260620 21:59:07.950235 12737 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6bff06caf3474692a4aebb2be266c3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42889 } }
I20260620 21:59:07.950410 12737 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:07.950448 12737 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6bff06caf3474692a4aebb2be266c3, State: Initialized, Role: FOLLOWER
I20260620 21:59:07.950639 12737 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6bff06caf3474692a4aebb2be266c3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42889 } }
I20260620 21:59:07.950739 12737 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:07.950773 12737 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:07.950825 12737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:07.951905 12737 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6bff06caf3474692a4aebb2be266c3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42889 } }
I20260620 21:59:07.952008 12737 leader_election.cc:302] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa6bff06caf3474692a4aebb2be266c3; no voters:
I20260620 21:59:07.952235 12737 leader_election.cc:288] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:07.952257 12742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:07.952369 12742 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [term 1 LEADER]: Becoming Leader. State: Replica: fa6bff06caf3474692a4aebb2be266c3, State: Running, Role: LEADER
I20260620 21:59:07.952453 12742 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6bff06caf3474692a4aebb2be266c3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42889 } }
I20260620 21:59:07.952521 12737 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:07.952847 12744 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa6bff06caf3474692a4aebb2be266c3. Latest consensus state: current_term: 1 leader_uuid: "fa6bff06caf3474692a4aebb2be266c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6bff06caf3474692a4aebb2be266c3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42889 } } }
I20260620 21:59:07.953069 12744 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:07.953007 12743 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa6bff06caf3474692a4aebb2be266c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6bff06caf3474692a4aebb2be266c3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42889 } } }
I20260620 21:59:07.953349 12743 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:07.953527 12751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:07.954049 12751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:07.956135 12751 catalog_manager.cc:1347] Generated new cluster ID: 6f18e37634a9455da84204fa1f5d2543
I20260620 21:59:07.956194 12751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 21:59:07 dist-test-slave-gl2g krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:07.961503 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:07.949444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.6.241.190:42889)
0620 21:59:07.949640 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:07.949650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:07.949685 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:59:07.950217 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:07.950226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:07.950285 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:07.950454 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:07.951244 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:07.952480 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:07.953584 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:07.954197 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:07.954647 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:07.957166 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:07.957170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:07.957171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:07.957183 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:07.959366 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:07.959762 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:07.959765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:07.959766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:07.959920 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:07.960121 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:07.960121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:07.960122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:07.960472 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:07.961035 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:07.961111 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 21:59:07.961266 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260620 21:59:07.961898 12751 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:07.962450 12751 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:07.967736 12751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa6bff06caf3474692a4aebb2be266c3: Generated new TSK 0
I20260620 21:59:07.967958 12751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:59:07.998487 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42889
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:08.084378 12763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:08.084645 12763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:08.084684 12763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:08.085912 12763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:08.085988 12763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:08.086036 12763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:08.087652 12763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:08.087939 12763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:08.088195 12763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:08.093554 12768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:08.093801 12771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:08.093777 12763 server_base.cc:1034] running on GCE node
W20260620 21:59:08.093961 12769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:08.094254 12763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:08.094514 12763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:08.095692 12763 hybrid_clock.cc:648] HybridClock initialized: now 1781992748095692 us; error 62 us; skew 500 ppm
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.098397 12763 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:59:08.098937 12763 webserver.cc:460] Webserver started at http://127.6.241.129:41311/ using document root <none> and password file <none>
I20260620 21:59:08.099174 12763 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:08.099246 12763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:08.099401 12763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:08.100515 12763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "354462b0a8f04d6089648e95825f5c18"
format_stamp: "Formatted at 2026-06-20 21:59:08 on dist-test-slave-gl2g"
server_key: "fcdc22f9374eea7e1c3bc132caa05229"
server_key_iv: "cb42476dc7bbaec11e4644a33ca972d1"
server_key_version: "encryptionkey@0"
I20260620 21:59:08.100911 12763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "354462b0a8f04d6089648e95825f5c18"
format_stamp: "Formatted at 2026-06-20 21:59:08 on dist-test-slave-gl2g"
server_key: "fcdc22f9374eea7e1c3bc132caa05229"
server_key_iv: "cb42476dc7bbaec11e4644a33ca972d1"
server_key_version: "encryptionkey@0"
I20260620 21:59:08.102362 12763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 21:59:08.103274 12778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:08.103472 12763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 21:59:08.103595 12763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "354462b0a8f04d6089648e95825f5c18"
format_stamp: "Formatted at 2026-06-20 21:59:08 on dist-test-slave-gl2g"
server_key: "fcdc22f9374eea7e1c3bc132caa05229"
server_key_iv: "cb42476dc7bbaec11e4644a33ca972d1"
server_key_version: "encryptionkey@0"
I20260620 21:59:08.103683 12763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:08.116472 12763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:08.119005 12763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:08.119179 12763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:08.119433 12763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:08.119751 12763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:08.119804 12763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:08.119848 12763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:08.119879 12763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:08.126834 12763 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:33023
I20260620 21:59:08.126921 12891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:33023 every 8 connection(s)
I20260620 21:59:08.127292 12763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:08.134236 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 12763
I20260620 21:59:08.134330 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:08.134529 7110 external_mini_cluster.cc:1428] Setting key d6f608d31d64c0543611eb18e08a7803
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:08.138948 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.128076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:43203 (local address 127.6.241.190:42889)
0620 21:59:08.128124 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:08.128127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:08.128394 (+ 267us) server_negotiation.cc:407] Connection header received
0620 21:59:08.130756 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:08.130758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:08.130776 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:08.130847 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:08.131486 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.131982 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.132959 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.133210 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.133264 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:08.135599 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:08.135600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:08.135600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:08.135610 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:08.137474 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.137953 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.137953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.137954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.138005 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.138288 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.138288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.138289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.138388 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:08.138437 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:08.138796 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 21:59:08.138879 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 21:59:08.139238 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.128070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42889 (local address 127.6.241.129:43203)
0620 21:59:08.128242 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:08.128287 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:08.130541 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:08.130914 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:08.130918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:08.131195 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:08.131398 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:08.131401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.132061 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.132061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:08.132886 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:08.132888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.133291 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.133292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:08.133402 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:08.133904 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:08.133914 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:08.135487 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:08.137572 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:08.137578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:08.137580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:08.137872 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:08.138070 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:08.138071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:08.138071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:08.138225 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:08.138451 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:08.138457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:08.138653 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 21:59:08.138765 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 21:59:08.138946 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260620 21:59:08.139772 12892 heartbeater.cc:347] Connected to a master server at 127.6.241.190:42889
I20260620 21:59:08.139883 12892 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:08.140085 12892 heartbeater.cc:511] Master 127.6.241.190:42889 requested a full tablet report, sending...
I20260620 21:59:08.140637 12701 ts_manager.cc:194] Registered new tserver with Master: 354462b0a8f04d6089648e95825f5c18 (127.6.241.129:33023)
I20260620 21:59:08.141260 12701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:43203
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:59:08.162470 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42889
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:08.246944 12897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:08.247174 12897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:08.247200 12897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:08.248361 12897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:08.248404 12897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:08.248430 12897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:08.249881 12897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:08.250099 12897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:08.250296 12897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:08.255574 12905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:08.255683 12902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:08.255735 12903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:08.256047 12897 server_base.cc:1034] running on GCE node
I20260620 21:59:08.256292 12897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:08.256577 12897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:08.257741 12897 hybrid_clock.cc:648] HybridClock initialized: now 1781992748257730 us; error 26 us; skew 500 ppm
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.260577 12897 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:59:08.261153 12897 webserver.cc:460] Webserver started at http://127.6.241.130:44947/ using document root <none> and password file <none>
I20260620 21:59:08.261416 12897 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:08.261487 12897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:08.261646 12897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:08.262830 12897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "fa0797889d174af9af311f120e578f46"
format_stamp: "Formatted at 2026-06-20 21:59:08 on dist-test-slave-gl2g"
server_key: "a154893d56ad1fc939dba430bbdaaca6"
server_key_iv: "57fe0be949e5d44880bbce525210c227"
server_key_version: "encryptionkey@0"
I20260620 21:59:08.263206 12897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "fa0797889d174af9af311f120e578f46"
format_stamp: "Formatted at 2026-06-20 21:59:08 on dist-test-slave-gl2g"
server_key: "a154893d56ad1fc939dba430bbdaaca6"
server_key_iv: "57fe0be949e5d44880bbce525210c227"
server_key_version: "encryptionkey@0"
I20260620 21:59:08.264552 12897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:08.265264 12912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:08.265447 12897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:08.265542 12897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "fa0797889d174af9af311f120e578f46"
format_stamp: "Formatted at 2026-06-20 21:59:08 on dist-test-slave-gl2g"
server_key: "a154893d56ad1fc939dba430bbdaaca6"
server_key_iv: "57fe0be949e5d44880bbce525210c227"
server_key_version: "encryptionkey@0"
I20260620 21:59:08.265627 12897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:08.275588 12897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:08.277863 12897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:08.278048 12897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:08.278328 12897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:08.278740 12897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:08.278797 12897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:08.278841 12897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:08.278868 12897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:08.284735 12897 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:44953
I20260620 21:59:08.284833 13025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:44953 every 8 connection(s)
I20260620 21:59:08.285203 12897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:08.287699 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 12897
I20260620 21:59:08.287825 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:08.288038 7110 external_mini_cluster.cc:1428] Setting key 8b7ea3177c8735e313f18e1a91f0868c
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:08.296033 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.286058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:37513 (local address 127.6.241.190:42889)
0620 21:59:08.286146 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:08.286148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:08.286259 (+ 111us) server_negotiation.cc:407] Connection header received
0620 21:59:08.287294 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:08.287295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:08.287307 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:08.287364 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:08.288173 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.288815 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.290004 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.290228 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.290315 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:08.292601 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:08.292602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:08.292603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:08.292612 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:08.294332 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.294873 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.294875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.294875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.294948 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.295288 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.295289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.295289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.295452 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:08.295525 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:08.295813 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 21:59:08.295912 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:59:08.296079 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.285956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42889 (local address 127.6.241.130:37513)
0620 21:59:08.286108 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:08.286163 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:08.287184 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:08.287427 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:08.287430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:08.287742 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:08.288022 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:08.288027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.288907 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.288908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:08.289911 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:08.289914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.290325 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.290326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:08.290457 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:08.291078 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:08.291087 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:08.292495 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:08.294438 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:08.294445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:08.294447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:08.294780 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:08.295041 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:08.295041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:08.295042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:08.295200 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:08.295520 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:08.295524 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:08.295662 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 21:59:08.295745 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 21:59:08.295880 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260620 21:59:08.296718 13026 heartbeater.cc:347] Connected to a master server at 127.6.241.190:42889
I20260620 21:59:08.296824 13026 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
I20260620 21:59:08.297032 13026 heartbeater.cc:511] Master 127.6.241.190:42889 requested a full tablet report, sending...
I20260620 21:59:08.297547 12701 ts_manager.cc:194] Registered new tserver with Master: fa0797889d174af9af311f120e578f46 (127.6.241.130:44953)
I20260620 21:59:08.297967 12701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:37513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:59:08.321825 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42889
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:08.405205 13031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:08.405426 13031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:08.405450 13031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:08.406771 13031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:08.406818 13031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:08.406876 13031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:08.408461 13031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:08.408674 13031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:08.408862 13031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:08.414026 13037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:08.413977 13036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:08.413964 13039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:08.414179 13031 server_base.cc:1034] running on GCE node
I20260620 21:59:08.414479 13031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:08.414757 13031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:08.415905 13031 hybrid_clock.cc:648] HybridClock initialized: now 1781992748415891 us; error 28 us; skew 500 ppm
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.418500 13031 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:59:08.419019 13031 webserver.cc:460] Webserver started at http://127.6.241.131:44247/ using document root <none> and password file <none>
I20260620 21:59:08.419263 13031 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:08.419332 13031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:08.419487 13031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:08.420604 13031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "20b32f8e7f51469caf789852ed09d148"
format_stamp: "Formatted at 2026-06-20 21:59:08 on dist-test-slave-gl2g"
server_key: "d4c23574b49a2ef4d63e13601fdc258c"
server_key_iv: "cf61164d3cda16e4a67b2502e64dd04c"
server_key_version: "encryptionkey@0"
I20260620 21:59:08.420975 13031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "20b32f8e7f51469caf789852ed09d148"
format_stamp: "Formatted at 2026-06-20 21:59:08 on dist-test-slave-gl2g"
server_key: "d4c23574b49a2ef4d63e13601fdc258c"
server_key_iv: "cf61164d3cda16e4a67b2502e64dd04c"
server_key_version: "encryptionkey@0"
I20260620 21:59:08.422488 13031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 21:59:08.423396 13046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:08.423743 13031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:08.423869 13031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "20b32f8e7f51469caf789852ed09d148"
format_stamp: "Formatted at 2026-06-20 21:59:08 on dist-test-slave-gl2g"
server_key: "d4c23574b49a2ef4d63e13601fdc258c"
server_key_iv: "cf61164d3cda16e4a67b2502e64dd04c"
server_key_version: "encryptionkey@0"
I20260620 21:59:08.423978 13031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:08.439620 13031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:08.442092 13031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:08.442260 13031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:08.442497 13031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:08.442842 13031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:08.442899 13031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:08.442935 13031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:08.442961 13031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:08.448840 13031 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:42407
I20260620 21:59:08.448881 13159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:42407 every 8 connection(s)
I20260620 21:59:08.449214 13031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:08.457365 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 13031
I20260620 21:59:08.457479 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:08.457690 7110 external_mini_cluster.cc:1428] Setting key fee81f5e9eb004defc14394a35f60fa6
I20260620 21:59:08.459563 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.449986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:57191 (local address 127.6.241.190:42889)
0620 21:59:08.450045 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:08.450047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:08.450263 (+ 216us) server_negotiation.cc:407] Connection header received
0620 21:59:08.451443 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:08.451444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:08.451455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:08.451523 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:08.452214 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.452725 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.453894 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.454090 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.454148 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:08.456547 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:08.456549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:08.456549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:08.456556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:08.458129 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.458581 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.458582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.458582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.458681 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.459005 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.459006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.459006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.459120 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:08.459176 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:08.459415 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 21:59:08.459497 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 21:59:08.459796 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.449943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42889 (local address 127.6.241.131:57191)
0620 21:59:08.450125 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:08.450174 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:08.451332 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:08.451601 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:08.451605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:08.451912 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:08.452123 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:08.452127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.452854 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.452854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:08.453818 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:08.453821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.454167 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.454168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:08.454288 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:08.454843 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:08.454852 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:08.456414 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:08.458220 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:08.458226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:08.458228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:08.458487 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:08.458832 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:08.458832 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:08.458833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:08.458942 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:08.459176 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:08.459180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:08.459302 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 21:59:08.459387 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:59:08.459556 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260620 21:59:08.460331 13160 heartbeater.cc:347] Connected to a master server at 127.6.241.190:42889
I20260620 21:59:08.460433 13160 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:08.460614 13160 heartbeater.cc:511] Master 127.6.241.190:42889 requested a full tablet report, sending...
I20260620 21:59:08.461030 12701 ts_manager.cc:194] Registered new tserver with Master: 20b32f8e7f51469caf789852ed09d148 (127.6.241.131:42407)
I20260620 21:59:08.461434 12701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:57191
I20260620 21:59:08.469765 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/test-admin.keytab.
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.485216 7110 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 21:59:08.487027 13167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:08.486169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.6.241.129:33023)
0620 21:59:08.486348 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:08.486353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:08.486362 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:08.486387 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:08.486389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:08.486403 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:08.486539 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:08.486940 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36192: BlockingRecv error: recv got EOF from 127.0.0.1:36192 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
W20260620 21:59:08.487113 7110 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.488176 7110 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.489166 7110 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:59:08.498163 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.489773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.6.241.129:33023)
0620 21:59:08.489846 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:08.489848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:08.489857 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:08.489906 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:08.489907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:08.489919 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:08.489989 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:08.490437 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.491247 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.491986 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.492333 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.492425 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:08.494864 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:08.494877 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:08.494877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:08.494891 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:08.496842 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.497297 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.497302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.497302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.497368 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.497575 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.497576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.497576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.497734 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:08.497898 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:08.497946 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 21:59:08.498029 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 21:59:08.500070 13167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:08.499559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.6.241.129:33023)
0620 21:59:08.499608 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:08.499611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:08.499635 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:59:08.499702 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:08.499702 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:08.499711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:08.499760 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:08.500002 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36202: BlockingRecv error: recv got EOF from 127.0.0.1:36202 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 21:59:08.500168 7110 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.501183 7110 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.502142 7110 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:59:08.509684 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.502730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.6.241.129:33023)
0620 21:59:08.502814 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:08.502816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:08.502864 (+ 48us) server_negotiation.cc:407] Connection header received
0620 21:59:08.502906 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:08.502906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:08.502916 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:08.502966 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:08.503353 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.503848 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.504512 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.504707 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.504760 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:08.507038 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:08.507040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:08.507041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:08.507051 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:08.508559 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.508910 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.508912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.508912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.508969 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.509170 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.509171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.509172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.509278 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:08.509381 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:08.509485 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 21:59:08.509568 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 21:59:08.511346 13167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:08.510814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.6.241.129:33023)
0620 21:59:08.510850 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:08.510851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:08.510873 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:59:08.510925 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:08.510926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:08.510933 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:08.510977 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:08.511298 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36214: BlockingRecv error: recv got EOF from 127.0.0.1:36214 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 21:59:08.511461 7110 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992718478901-7110-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.512503 7110 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:08.513420 7110 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 21:59:08.520308 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.513981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.6.241.129:33023)
0620 21:59:08.514043 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:08.514045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:08.514055 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:08.514135 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:08.514136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:08.514144 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:08.514191 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:08.514544 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.515137 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.515784 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:08.515972 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:08.516025 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:08.518020 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:08.518021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:08.518022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:08.518027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:08.519468 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.519755 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.519756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.519756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.519803 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:08.519942 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:08.519942 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:08.519942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:08.520023 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:08.520089 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:08.520189 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 21:59:08.520242 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:59:08.520979 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 12763
I20260620 21:59:08.525939 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 12897
I20260620 21:59:08.530727 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 13031
I20260620 21:59:08.535610 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 12669
2026-06-20T21:59:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1903 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[13180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[13180](info): set up 2 sockets
Jun 20 21:59:08 dist-test-slave-gl2g krb5kdc[13180](info): commencing operation
krb5kdc: starting...
W20260620 21:59:09.571141 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:59:09 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.190.keytab.
Entry for principal oryx/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.190.keytab.
I20260620 21:59:09.695228 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:43375
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.190.keytab
--principal=oryx/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:43375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:09.777734 13193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:09.777971 13193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:09.777994 13193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:09.779171 13193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:09.779212 13193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:09.779227 13193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:09.779240 13193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:09.779253 13193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:09.780694 13193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:43375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.190.keytab
--principal=oryx/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:43375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:09.780913 13193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:09.781105 13193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:09.786159 13199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:09.786145 13198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:09.786145 13201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:09.786290 13193 server_base.cc:1034] running on GCE node
I20260620 21:59:09.786684 13193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:09.786970 13193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:09.788151 13193 hybrid_clock.cc:648] HybridClock initialized: now 1781992749788136 us; error 32 us; skew 500 ppm
Jun 20 21:59:09 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992749, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:09.794420 13193 init.cc:377] Logged in from keytab as oryx/127.6.241.190@KRBTEST.COM (short username oryx)
I20260620 21:59:09.794983 13193 webserver.cc:460] Webserver started at http://127.6.241.190:34137/ using document root <none> and password file <none>
I20260620 21:59:09.795233 13193 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:09.795301 13193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:09.795444 13193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:09.796535 13193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "782f2eef2b6e43c3a1f1122b1598d935"
format_stamp: "Formatted at 2026-06-20 21:59:09 on dist-test-slave-gl2g"
server_key: "9fd99113398dc75fef34fb36a15e9258"
server_key_iv: "56038aa4a145c879e4e9005339db60d9"
server_key_version: "encryptionkey@0"
I20260620 21:59:09.796905 13193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "782f2eef2b6e43c3a1f1122b1598d935"
format_stamp: "Formatted at 2026-06-20 21:59:09 on dist-test-slave-gl2g"
server_key: "9fd99113398dc75fef34fb36a15e9258"
server_key_iv: "56038aa4a145c879e4e9005339db60d9"
server_key_version: "encryptionkey@0"
I20260620 21:59:09.798238 13193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:09.799022 13208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:09.799216 13193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:09.799329 13193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "782f2eef2b6e43c3a1f1122b1598d935"
format_stamp: "Formatted at 2026-06-20 21:59:09 on dist-test-slave-gl2g"
server_key: "9fd99113398dc75fef34fb36a15e9258"
server_key_iv: "56038aa4a145c879e4e9005339db60d9"
server_key_version: "encryptionkey@0"
I20260620 21:59:09.799407 13193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:09.818321 13193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:09.820533 13193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:09.820727 13193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:09.824829 13193 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:43375
I20260620 21:59:09.824867 13260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:43375 every 8 connection(s)
I20260620 21:59:09.825325 13193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:09.826339 13261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:09.828748 13261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: Bootstrap starting.
I20260620 21:59:09.829348 13261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:09.829643 13261 log.cc:824] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:09.830224 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 13193
I20260620 21:59:09.830339 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:09.830497 13261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: No bootstrap required, opened a new log
I20260620 21:59:09.830536 7110 external_mini_cluster.cc:1428] Setting key b5f3bb3913a7ed75c51ed11c8b74b872
I20260620 21:59:09.832530 13261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } }
I20260620 21:59:09.832710 13261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:09.832758 13261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782f2eef2b6e43c3a1f1122b1598d935, State: Initialized, Role: FOLLOWER
I20260620 21:59:09.832908 13261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } }
I20260620 21:59:09.832986 13261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:09.833029 13261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:09.833077 13261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:09.833676 13261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } }
I20260620 21:59:09.833792 13261 leader_election.cc:302] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 782f2eef2b6e43c3a1f1122b1598d935; no voters:
I20260620 21:59:09.833977 13261 leader_election.cc:288] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:09.833999 13266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:09.834121 13266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 1 LEADER]: Becoming Leader. State: Replica: 782f2eef2b6e43c3a1f1122b1598d935, State: Running, Role: LEADER
I20260620 21:59:09.834204 13266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } }
I20260620 21:59:09.834249 13261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:09.834559 13268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } }
I20260620 21:59:09.834545 13266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782f2eef2b6e43c3a1f1122b1598d935. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } }
I20260620 21:59:09.834780 13266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:09.834971 13268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:09.834991 13273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:09.835536 13273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:09.837980 13283 mvcc.cc:205] Tried to move back new op lower bound from 7299042303330197504 to 7299042303321272320. Current Snapshot: MvccSnapshot[applied={T|T < 7299042303330197504}]
I20260620 21:59:09.838465 13273 catalog_manager.cc:1347] Generated new cluster ID: 64f670dccdbc4b8cbbf599ac69360f5d
I20260620 21:59:09.838526 13273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 21:59:09 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:09.843055 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:09.831154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.6.241.190:43375)
0620 21:59:09.831361 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:09.831371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:09.831406 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:59:09.831847 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:09.831854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:09.831904 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:09.832075 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:09.832734 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:09.834128 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:09.835269 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:09.836075 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:09.836188 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:09.839118 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:09.839122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:09.839123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:09.839137 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:09.841292 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:09.841696 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:09.841700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:09.841700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:09.841817 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:09.842031 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:09.842031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:09.842032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:09.842365 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:09.842530 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:09.842668 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 21:59:09.842820 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260620 21:59:09.850212 13273 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:09.850879 13273 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:09.853778 13273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: Generated new TSK 0
I20260620 21:59:09.853971 13273 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab.
Entry for principal oryx/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab.
I20260620 21:59:09.887017 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43375
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab
--principal=oryx/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:09.982391 13287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:09.982677 13287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:09.982733 13287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:09.984050 13287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:09.984100 13287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:09.984151 13287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:09.985666 13287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab
--principal=oryx/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:09.985929 13287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:09.986187 13287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:09.991443 13292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:09.991437 13293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:09.991562 13287 server_base.cc:1034] running on GCE node
W20260620 21:59:09.991434 13295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:09.991890 13287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:09.992188 13287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:09.993331 13287 hybrid_clock.cc:648] HybridClock initialized: now 1781992749993319 us; error 28 us; skew 500 ppm
Jun 20 21:59:09 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992749, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:09.995962 13287 init.cc:377] Logged in from keytab as oryx/127.6.241.129@KRBTEST.COM (short username oryx)
I20260620 21:59:09.996443 13287 webserver.cc:460] Webserver started at http://127.6.241.129:35585/ using document root <none> and password file <none>
I20260620 21:59:09.996675 13287 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:09.996742 13287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:09.996891 13287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:09.997983 13287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "da9da6d6ed39410d914401852c1b60a5"
format_stamp: "Formatted at 2026-06-20 21:59:09 on dist-test-slave-gl2g"
server_key: "55b6daab21c60f4d9964183f35a19db3"
server_key_iv: "ed0ff48e99f3a5e700441a650c3ba0f2"
server_key_version: "encryptionkey@0"
I20260620 21:59:09.998350 13287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "da9da6d6ed39410d914401852c1b60a5"
format_stamp: "Formatted at 2026-06-20 21:59:09 on dist-test-slave-gl2g"
server_key: "55b6daab21c60f4d9964183f35a19db3"
server_key_iv: "ed0ff48e99f3a5e700441a650c3ba0f2"
server_key_version: "encryptionkey@0"
I20260620 21:59:09.999804 13287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 21:59:10.000614 13302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.000808 13287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:10.000905 13287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "da9da6d6ed39410d914401852c1b60a5"
format_stamp: "Formatted at 2026-06-20 21:59:09 on dist-test-slave-gl2g"
server_key: "55b6daab21c60f4d9964183f35a19db3"
server_key_iv: "ed0ff48e99f3a5e700441a650c3ba0f2"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.000988 13287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.012809 13287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:10.015233 13287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:10.015401 13287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:10.015648 13287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:10.015952 13287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:10.016018 13287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.016062 13287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:10.016093 13287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.021875 13287 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:44265
I20260620 21:59:10.021921 13415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:44265 every 8 connection(s)
I20260620 21:59:10.022257 13287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:10.022753 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 13287
I20260620 21:59:10.022866 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:10.023092 7110 external_mini_cluster.cc:1428] Setting key 7f9cf0810bec2567b34e32151f8bb799
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992749, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.241.130@KRBTEST.COM; defaulting to no policy
I20260620 21:59:10.032922 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.023040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:43343 (local address 127.6.241.190:43375)
0620 21:59:10.023109 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.023112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.023268 (+ 156us) server_negotiation.cc:407] Connection header received
0620 21:59:10.023935 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.023936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.023948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.023993 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.025039 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.025875 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.027028 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.027396 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.027480 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.029874 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.029876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.029877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.029887 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.031617 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.032022 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.032023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.032024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.032077 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.032316 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.032316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.032316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.032426 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.032478 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.032745 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.032841 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:59:10.033105 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.022969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.129:43343)
0620 21:59:10.023138 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.023182 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.023833 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.024069 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.024073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.024678 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.024911 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.024918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.025995 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.025996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.026938 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.026940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.027482 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.027483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.027585 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.028135 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.028146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.029773 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.031702 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.031709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.031711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.031948 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.032139 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.032140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.032140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.032261 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.032483 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.032488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.032620 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.032717 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.032848 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260620 21:59:10.033676 13416 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43375
I20260620 21:59:10.033768 13416 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:10.033998 13416 heartbeater.cc:511] Master 127.6.241.190:43375 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.130@KRBTEST.COM" created.
I20260620 21:59:10.034672 13225 ts_manager.cc:194] Registered new tserver with Master: da9da6d6ed39410d914401852c1b60a5 (127.6.241.129:44265)
I20260620 21:59:10.035674 13225 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:43343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab.
Entry for principal oryx/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab.
I20260620 21:59:10.047689 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43375
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab
--principal=oryx/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:10.134194 13421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:10.134500 13421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:10.134541 13421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:10.136524 13421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:10.136579 13421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:10.136638 13421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:10.138502 13421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab
--principal=oryx/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:10.138835 13421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:10.139101 13421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:10.144212 13426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.144375 13427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.144448 13421 server_base.cc:1034] running on GCE node
W20260620 21:59:10.144532 13429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.144768 13421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:10.144986 13421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:10.146181 13421 hybrid_clock.cc:648] HybridClock initialized: now 1781992750146168 us; error 30 us; skew 500 ppm
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:10.148814 13421 init.cc:377] Logged in from keytab as oryx/127.6.241.130@KRBTEST.COM (short username oryx)
I20260620 21:59:10.149327 13421 webserver.cc:460] Webserver started at http://127.6.241.130:40757/ using document root <none> and password file <none>
I20260620 21:59:10.149572 13421 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:10.149641 13421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:10.149792 13421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:10.150969 13421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "93e1b09488874feab266f10e487666cc"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "c50ae6a5bc686e9d5da7a241c58c034a"
server_key_iv: "e84740a2706b1217d108803390102893"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.151372 13421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "93e1b09488874feab266f10e487666cc"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "c50ae6a5bc686e9d5da7a241c58c034a"
server_key_iv: "e84740a2706b1217d108803390102893"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.152820 13421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:10.153594 13436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.153782 13421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:10.153885 13421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "93e1b09488874feab266f10e487666cc"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "c50ae6a5bc686e9d5da7a241c58c034a"
server_key_iv: "e84740a2706b1217d108803390102893"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.153963 13421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.161619 13421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:10.163851 13421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:10.164021 13421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:10.164268 13421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:10.164578 13421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:10.164630 13421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.164672 13421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:10.164698 13421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.170583 13421 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:44769
I20260620 21:59:10.170694 13549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:44769 every 8 connection(s)
I20260620 21:59:10.171023 13421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:10.172650 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 13421
I20260620 21:59:10.172753 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:10.172926 7110 external_mini_cluster.cc:1428] Setting key ef20cc8f964244b7778d886befa62960
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.241.131@KRBTEST.COM; defaulting to no policy
I20260620 21:59:10.180898 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.171782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:60491 (local address 127.6.241.190:43375)
0620 21:59:10.171828 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.171831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.172024 (+ 193us) server_negotiation.cc:407] Connection header received
0620 21:59:10.172575 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.172576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.172590 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.172642 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.173802 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.174305 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.175376 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.175574 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.175656 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.178014 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.178016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.178017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.178025 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.179490 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.179893 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.179894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.179895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.179955 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.180212 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.180213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.180213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.180331 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.180422 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.180715 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.180803 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 21:59:10.181102 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.171746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.130:60491)
0620 21:59:10.171893 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.171937 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.172455 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.172721 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.172725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.173468 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.173715 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.173719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.174404 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.174405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.175289 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.175292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.175672 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.175673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.175799 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.176356 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.176365 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.177919 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.179583 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.179590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.179592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.179820 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.180020 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.180021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.180022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.180132 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.180415 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.180421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.180566 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.180693 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.180856 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260620 21:59:10.181589 13550 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43375
I20260620 21:59:10.181720 13550 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:10.181948 13550 heartbeater.cc:511] Master 127.6.241.190:43375 requested a full tablet report, sending...
I20260620 21:59:10.182336 13225 ts_manager.cc:194] Registered new tserver with Master: 93e1b09488874feab266f10e487666cc (127.6.241.130:44769)
I20260620 21:59:10.182798 13225 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:60491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab.
Entry for principal oryx/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab.
I20260620 21:59:10.197283 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43375
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab
--principal=oryx/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:10.279582 13555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:10.279847 13555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:10.279884 13555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:10.281095 13555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:10.281165 13555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:10.281210 13555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:10.282744 13555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab
--principal=oryx/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:10.283021 13555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:10.283257 13555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:10.288539 13563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.288552 13561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.288535 13555 server_base.cc:1034] running on GCE node
W20260620 21:59:10.288582 13560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.289062 13555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:10.289343 13555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:10.290536 13555 hybrid_clock.cc:648] HybridClock initialized: now 1781992750290521 us; error 39 us; skew 500 ppm
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:10.293237 13555 init.cc:377] Logged in from keytab as oryx/127.6.241.131@KRBTEST.COM (short username oryx)
I20260620 21:59:10.293655 13555 webserver.cc:460] Webserver started at http://127.6.241.131:43295/ using document root <none> and password file <none>
I20260620 21:59:10.293848 13555 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:10.293890 13555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:10.293989 13555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:10.295113 13555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "60e8765db7344cf180a4df768e8623be"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "6198210a6ea17597b4e83fec2f3a0a4d"
server_key_iv: "f9db6a341b455092930d0e6db217a104"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.295421 13555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "60e8765db7344cf180a4df768e8623be"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "6198210a6ea17597b4e83fec2f3a0a4d"
server_key_iv: "f9db6a341b455092930d0e6db217a104"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.296715 13555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:10.297421 13570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.297616 13555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:10.297706 13555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "60e8765db7344cf180a4df768e8623be"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "6198210a6ea17597b4e83fec2f3a0a4d"
server_key_iv: "f9db6a341b455092930d0e6db217a104"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.297782 13555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.312482 13555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:10.314482 13555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:10.314656 13555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:10.314903 13555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:10.315238 13555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:10.315280 13555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.315303 13555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:10.315315 13555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.321177 13555 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:41127
I20260620 21:59:10.321211 13683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:41127 every 8 connection(s)
I20260620 21:59:10.321547 13555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:10.322142 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 13555
I20260620 21:59:10.322258 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:10.322476 7110 external_mini_cluster.cc:1428] Setting key 4bb20b20448b5fbd9ec215c605102067
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:10.331177 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.322326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:39721 (local address 127.6.241.190:43375)
0620 21:59:10.322383 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.322386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.322652 (+ 266us) server_negotiation.cc:407] Connection header received
0620 21:59:10.323306 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.323307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.323320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.323360 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.324339 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.324858 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.325892 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.326129 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.326205 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.328528 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.328530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.328531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.328537 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.329791 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.330209 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.330210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.330211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.330272 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.330559 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.330560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.330560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.330718 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.330773 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.331029 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.331095 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:59:10.331400 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.322329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:39721)
0620 21:59:10.322504 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.322547 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.323206 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.323441 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.323445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.324035 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.324239 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.324244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.324954 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.324955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.325780 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.325782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.326212 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.326213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.326354 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.326957 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.326966 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.328425 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.329865 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.329871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.329872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.330115 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.330363 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.330364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.330365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.330481 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.330779 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.330784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.330913 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.331008 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.331143 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260620 21:59:10.331948 13684 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43375
I20260620 21:59:10.332060 13684 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:10.332216 13684 heartbeater.cc:511] Master 127.6.241.190:43375 requested a full tablet report, sending...
I20260620 21:59:10.332602 13225 ts_manager.cc:194] Registered new tserver with Master: 60e8765db7344cf180a4df768e8623be (127.6.241.131:41127)
I20260620 21:59:10.332973 13225 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:39721
I20260620 21:59:10.334198 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.241.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.189.keytab.
Entry for principal oryx/127.6.241.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.189.keytab.
I20260620 21:59:10.360785 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.189:32981
--webserver_interface=127.6.241.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.189.keytab
--principal=oryx/127.6.241.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:43375,127.6.241.189:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:10.443874 13689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:10.444113 13689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:10.444136 13689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:10.445375 13689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:10.445420 13689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:10.445434 13689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:10.445447 13689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:10.445459 13689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:10.446943 13689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:43375,127.6.241.189:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.189.keytab
--principal=oryx/127.6.241.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.189:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.241.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
W20260620 21:59:10.447043 13689 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.241.190:43375,127.6.241.189:32981'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 21:59:10.447170 13689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:10.447360 13689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:10.452363 13689 server_base.cc:1034] running on GCE node
W20260620 21:59:10.452307 13695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.452322 13697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.452332 13694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.452796 13689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:10.453114 13689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:10.454293 13689 hybrid_clock.cc:648] HybridClock initialized: now 1781992750454281 us; error 32 us; skew 500 ppm
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:10.457036 13689 init.cc:377] Logged in from keytab as oryx/127.6.241.189@KRBTEST.COM (short username oryx)
I20260620 21:59:10.457520 13689 webserver.cc:460] Webserver started at http://127.6.241.189:45125/ using document root <none> and password file <none>
I20260620 21:59:10.457783 13689 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:10.457831 13689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:10.457999 13689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:10.459203 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data/instance:
uuid: "d4d0aab8261446af860d68672e92d0de"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "8dcc0729d7f9d624c5afd0086425b1ea"
server_key_iv: "0b2777a93d7de22be9d89d82bc0c0c8a"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.459578 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal/instance:
uuid: "d4d0aab8261446af860d68672e92d0de"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "8dcc0729d7f9d624c5afd0086425b1ea"
server_key_iv: "0b2777a93d7de22be9d89d82bc0c0c8a"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.460996 13689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:10.461844 13704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.462013 13689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:10.462102 13689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal
uuid: "d4d0aab8261446af860d68672e92d0de"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "8dcc0729d7f9d624c5afd0086425b1ea"
server_key_iv: "0b2777a93d7de22be9d89d82bc0c0c8a"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.462184 13689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.479990 13689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:10.482218 13689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:10.482395 13689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:10.486410 13689 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.189:32981
I20260620 21:59:10.486485 13756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.189:32981 every 8 connection(s)
I20260620 21:59:10.486840 13689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
I20260620 21:59:10.487805 13757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:10.489401 13757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.189@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:10.496099 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 13689
I20260620 21:59:10.499202 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.490136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.6.241.190:43375)
0620 21:59:10.490200 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.490202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.490419 (+ 217us) server_negotiation.cc:407] Connection header received
0620 21:59:10.490988 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.490989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.491002 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.491055 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.491716 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.492223 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.493231 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.493423 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.493504 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.495725 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.495727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.495727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.495735 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.497525 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.498097 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.498098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.498098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.498153 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.498434 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.498435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.498435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.498583 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.498711 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.499017 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.499105 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 21:59:10.499316 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.490100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:59964)
0620 21:59:10.490292 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.490329 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.490878 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.491138 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.491142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.491406 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.491600 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.491604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.492314 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.492315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.493139 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.493141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.493511 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.493512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.493645 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.494165 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.494174 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.495640 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.497625 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.497631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.497634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.497980 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.498219 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.498220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.498220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.498366 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.498707 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.498712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.498879 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.498979 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.499115 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260620 21:59:10.499998 13757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.241.189@KRBTEST.COM
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.189@KRBTEST.COM for oryx/127.6.241.189@KRBTEST.COM
I20260620 21:59:10.506580 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.496608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.6.241.189:32981)
0620 21:59:10.496818 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.496824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.496853 (+ 29us) server_negotiation.cc:407] Connection header received
0620 21:59:10.496924 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.496926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.496938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.496991 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.497746 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.498801 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.499843 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.500204 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.500289 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.502787 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.502795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.502796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.502802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.504859 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.505243 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.505247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.505248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.505310 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.505547 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.505548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.505549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.505830 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.506216 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.506437 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.506502 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":99,"threads_started":1}
I20260620 21:59:10.508221 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.500284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.0.0.1:52000)
0620 21:59:10.500381 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.500390 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.500418 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.500855 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.500856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.500976 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.501018 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.501020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.501762 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.501763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.502331 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.502333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.502757 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.502758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.502826 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.503295 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.503299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.504803 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.507426 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.507427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.507427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.507608 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.507778 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.507778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.507779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.507824 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.507989 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.507990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.508023 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.508071 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.508141 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 21:59:10.508293 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.500329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.6.241.189:32981)
0620 21:59:10.500555 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.500557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.500565 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:10.500584 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.500585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.500594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.500660 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.501103 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.501678 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.502395 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.502674 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.502757 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.504897 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.504898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.504898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.504904 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.507323 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.507674 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.507675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.507675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.507723 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.507863 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.507864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.507864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.507945 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.507988 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.508087 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.508186 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260620 21:59:10.509466 13757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: Bootstrap starting.
I20260620 21:59:10.510047 13757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:10.510273 13757 log.cc:824] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:10.510963 13757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: No bootstrap required, opened a new log
I20260620 21:59:10.512308 13757 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } }
I20260620 21:59:10.512434 13757 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:10.512455 13757 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d0aab8261446af860d68672e92d0de, State: Initialized, Role: FOLLOWER
I20260620 21:59:10.512583 13757 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } }
I20260620 21:59:10.512821 13764 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } } }
I20260620 21:59:10.512900 13757 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:10.512905 13764 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: This master's current role is: FOLLOWER
W20260620 21:59:10.514386 13775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:10.514442 13775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:10.515935 13689 master_runner.cc:428] Detected that this master d4d0aab8261446af860d68672e92d0de is joining an existing cluster
I20260620 21:59:10.516006 13689 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.241.189:32981
I20260620 21:59:10.523803 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.517009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.6.241.190:43375)
0620 21:59:10.517050 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.517052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.517205 (+ 153us) server_negotiation.cc:407] Connection header received
0620 21:59:10.517306 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.517307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.517318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.517347 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.517789 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.518523 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.519338 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.519535 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.519614 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.520744 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.520746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.520746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.520753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.522436 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.522957 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.522959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.522960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.523029 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.523241 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.523242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.523243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.523381 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.523461 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.523610 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.523698 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 21:59:10.523820 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.516955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:59974)
0620 21:59:10.517138 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.517149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.517233 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.517408 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.517409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.517639 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.517710 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.517712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.518586 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.518587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.519247 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.519249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.519613 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.519614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.519685 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.520169 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.520175 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.520619 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.522537 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.522538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.522538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.522848 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.523104 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.523105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.523105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.523170 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.523451 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.523452 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.523502 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.523573 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.523675 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":49,"thread_start_us":78,"threads_started":1}
I20260620 21:59:10.524755 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.516956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.0.0.1:52008)
0620 21:59:10.517160 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.517179 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.517233 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.517354 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.517355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.517545 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.517608 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.517610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.518536 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.518538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.519545 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.519548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.520060 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.520061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.520118 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.520552 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.520557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.520938 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.523437 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.523438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.523438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.523633 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.523820 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.523820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.523821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.524118 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.524403 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.524404 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.524447 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.524516 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.524616 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":52,"spinlock_wait_cycles":8960,"thread_start_us":117,"threads_started":1}
I20260620 21:59:10.524819 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.517135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.6.241.189:32981)
0620 21:59:10.517169 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.517171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.517219 (+ 48us) server_negotiation.cc:407] Connection header received
0620 21:59:10.517288 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.517288 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.517297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.517350 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.517684 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.518452 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.519640 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.519948 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.519991 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.521023 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.521024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.521025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.521031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.523345 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.523716 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.523717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.523717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.523760 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.524199 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.524200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.524200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.524340 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.524411 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.524552 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.524689 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":20}
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992749, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.190@KRBTEST.COM for oryx/127.6.241.189@KRBTEST.COM
I20260620 21:59:10.533152 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.526218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.6.241.189:32981)
0620 21:59:10.526248 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.526249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.526399 (+ 150us) server_negotiation.cc:407] Connection header received
0620 21:59:10.526489 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.526490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.526500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.526547 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.527084 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.527542 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.528266 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.528466 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.528547 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.530715 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.530716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.530716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.530722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.531969 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.532380 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.532382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.532382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.532435 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.532631 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.532632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.532632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.532747 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.532818 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.532986 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.533064 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 21:59:10.533190 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.526162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.0.0.1:52014)
0620 21:59:10.526310 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.526342 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.526424 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.526550 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.526553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.526895 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.526952 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.526954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.527615 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.527616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.528186 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.528188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.528548 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.528549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.528633 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.529059 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.529063 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.530585 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.532065 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.532072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.532075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.532296 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.532511 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.532512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.532512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.532574 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.532809 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.532816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.532869 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.532963 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.533078 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260620 21:59:10.534191 13225 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.241.189:32981: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d4d0aab8261446af860d68672e92d0de"
member_type: NON_VOTER
last_known_addr {
host: "127.6.241.189"
port: 32981
}
attrs {
promote: true
}
}
dest_uuid: "782f2eef2b6e43c3a1f1122b1598d935"
cas_config_opid_index: -1
I20260620 21:59:10.534560 13225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } }
I20260620 21:59:10.535111 13785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d4d0aab8261446af860d68672e92d0de (127.6.241.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } } }
I20260620 21:59:10.535449 13731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:10.536144 13785 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.241.189:32981
I20260620 21:59:10.536365 13689 master.cc:537] Master@127.6.241.189:32981 shutting down...
I20260620 21:59:10.536428 13731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Refusing update from remote peer 782f2eef2b6e43c3a1f1122b1598d935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 21:59:10.536407 13784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } } }
I20260620 21:59:10.536486 13784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:10.536877 13784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:59:10.537642 13211 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 21:59:10.538029 13211 consensus_peers.cc:487] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 -> Peer d4d0aab8261446af860d68672e92d0de (127.6.241.189:32981): Couldn't send request to peer d4d0aab8261446af860d68672e92d0de. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:59:10.539366 13689 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:10.539461 13689 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:10.539513 13689 tablet_replica.cc:330] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: stopping tablet replica
I20260620 21:59:10.552603 13689 master.cc:552] Master@127.6.241.189:32981 shutdown complete.
I20260620 21:59:10.552662 13689 master_runner.cc:305] Clearing existing system tablet
I20260620 21:59:10.552819 13689 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:10.552871 13689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:10.553570 13689 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260620 21:59:10.554152 13789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.554281 13689 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.554339 13689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.554363 13689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal
uuid: "d4d0aab8261446af860d68672e92d0de"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "8dcc0729d7f9d624c5afd0086425b1ea"
server_key_iv: "0b2777a93d7de22be9d89d82bc0c0c8a"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.555153 13689 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:10.556809 13689 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 21:59:10.556941 13689 log.cc:1196] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 21:59:10.557179 13689 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: Deleting consensus metadata
I20260620 21:59:10.557315 13689 master_runner.cc:315] Copying system tablet from 127.6.241.190:43375
I20260620 21:59:10.558046 13689 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: tablet copy: Beginning tablet copy session from remote peer at address 127.6.241.190:43375
I20260620 21:59:10.565119 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.558482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.6.241.190:43375)
0620 21:59:10.558542 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.558545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.558747 (+ 202us) server_negotiation.cc:407] Connection header received
0620 21:59:10.558820 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.558821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.558831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.558888 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.559288 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.559795 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.560893 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.561085 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.561146 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.562240 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.562241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.562241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.562247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.563869 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.564295 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.564296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.564296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.564360 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.564604 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.564605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.564605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.564735 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.564803 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.564970 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.565040 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:59:10.565188 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.558377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:59976)
0620 21:59:10.558676 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.558686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.558754 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.558896 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.558897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.559108 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.559184 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.559186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.559883 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.559884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.560796 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.560799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.561162 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.561163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.561232 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.561725 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.561730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.562157 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.563952 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.563954 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.563954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.564199 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.564432 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.564433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.564434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.564502 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.564806 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.564807 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.564860 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.564952 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.565074 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":69,"threads_started":1}
I20260620 21:59:10.565551 13245 tablet_copy_service.cc:140] P 782f2eef2b6e43c3a1f1122b1598d935: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d4d0aab8261446af860d68672e92d0de ({username='oryx', principal='oryx/127.6.241.189@KRBTEST.COM'} at 127.0.0.1:59976)
I20260620 21:59:10.565668 13245 tablet_copy_service.cc:161] P 782f2eef2b6e43c3a1f1122b1598d935: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d4d0aab8261446af860d68672e92d0de at {username='oryx', principal='oryx/127.6.241.189@KRBTEST.COM'} at 127.0.0.1:59976: session id = d4d0aab8261446af860d68672e92d0de-00000000000000000000000000000000
I20260620 21:59:10.566550 13245 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 21:59:10.567276 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 21:59:10.568349 13689 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: tablet copy: Starting download of 0 data blocks...
I20260620 21:59:10.568519 13689 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: tablet copy: Starting download of 1 WAL segments...
I20260620 21:59:10.569598 13689 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 21:59:10.570586 13245 tablet_copy_service.cc:342] P 782f2eef2b6e43c3a1f1122b1598d935: Request end of tablet copy session d4d0aab8261446af860d68672e92d0de-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.241.189@KRBTEST.COM'} at 127.0.0.1:59976
I20260620 21:59:10.570698 13245 tablet_copy_service.cc:433] P 782f2eef2b6e43c3a1f1122b1598d935: ending tablet copy session d4d0aab8261446af860d68672e92d0de-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d4d0aab8261446af860d68672e92d0de
I20260620 21:59:10.573453 13689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:10.574731 13796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.574772 13797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.574875 13689 server_base.cc:1034] running on GCE node
W20260620 21:59:10.575094 13799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.575232 13689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:10.575363 13689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:10.576500 13689 hybrid_clock.cc:648] HybridClock initialized: now 1781992750576481 us; error 37 us; skew 500 ppm
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:10.578096 13689 init.cc:377] Logged in from keytab as oryx/127.6.241.189@KRBTEST.COM (short username oryx)
I20260620 21:59:10.578389 13689 webserver.cc:460] Webserver started at http://127.6.241.189:44621/ using document root <none> and password file <none>
I20260620 21:59:10.578528 13689 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:10.578572 13689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:10.579255 13689 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.579768 13806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.579883 13689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 21:59:10.579929 13689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal
uuid: "d4d0aab8261446af860d68672e92d0de"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "8dcc0729d7f9d624c5afd0086425b1ea"
server_key_iv: "0b2777a93d7de22be9d89d82bc0c0c8a"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.579991 13689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.585070 13689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:10.586905 13689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:10.590042 13689 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.189:32981
I20260620 21:59:10.590514 13689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
I20260620 21:59:10.590548 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.189:32981 every 8 connection(s)
I20260620 21:59:10.591492 13859 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:59:10.592031 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: Bootstrap starting.
I20260620 21:59:10.594954 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:59:10.595209 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: Bootstrap complete.
I20260620 21:59:10.595425 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } }
I20260620 21:59:10.595520 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4d0aab8261446af860d68672e92d0de, State: Initialized, Role: LEARNER
I20260620 21:59:10.595559 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } }
I20260620 21:59:10.595758 13862 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } } }
I20260620 21:59:10.595770 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:10.595872 13862 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: This master's current role is: LEARNER
I20260620 21:59:10.597381 13873 catalog_manager.cc:1259] Loaded cluster ID: 64f670dccdbc4b8cbbf599ac69360f5d
I20260620 21:59:10.597704 13873 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: loading cluster ID for follower catalog manager: success
I20260620 21:59:10.598544 13873 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: acquiring CA information for follower catalog manager: success
I20260620 21:59:10.598881 13873 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:59:10.615283 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.608534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52034 (local address 127.6.241.189:32981)
0620 21:59:10.608683 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.608684 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.608692 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:10.608715 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.608715 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.608725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.608798 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.609385 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.610057 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.610849 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.611057 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.611131 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.612482 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.612484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.612484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.612491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.614026 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.614403 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.614405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.614405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.614463 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.614717 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.614718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.614718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.614828 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.614929 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.615066 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.615157 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.6.241.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.188.keytab.
Entry for principal oryx/127.6.241.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.188.keytab.
I20260620 21:59:10.638957 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.188:46449
--webserver_interface=127.6.241.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.188.keytab
--principal=oryx/127.6.241.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:43375,127.6.241.189:32981,127.6.241.188:46449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:10.723614 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:10.723834 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:10.723857 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:10.725065 13877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:10.725116 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:10.725131 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:10.725145 13877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:10.725157 13877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:10.726751 13877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:43375,127.6.241.189:32981,127.6.241.188:46449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.188.keytab
--principal=oryx/127.6.241.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.188:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.241.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:10.727042 13877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:10.727432 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:10.732569 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.732564 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.732564 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.732599 13877 server_base.cc:1034] running on GCE node
I20260620 21:59:10.733073 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:10.733438 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:10.734668 13877 hybrid_clock.cc:648] HybridClock initialized: now 1781992750734648 us; error 44 us; skew 500 ppm
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:10.737703 13877 init.cc:377] Logged in from keytab as oryx/127.6.241.188@KRBTEST.COM (short username oryx)
I20260620 21:59:10.738220 13877 webserver.cc:460] Webserver started at http://127.6.241.188:36557/ using document root <none> and password file <none>
I20260620 21:59:10.738451 13877 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:10.738497 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:10.738687 13877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:10.739809 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data/instance:
uuid: "9543330b81d8478990d6e37386518b81"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "5e46bd1ef08055726749f611a80c8cce"
server_key_iv: "10745b8e33617240c339850dfca7b537"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.740187 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal/instance:
uuid: "9543330b81d8478990d6e37386518b81"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "5e46bd1ef08055726749f611a80c8cce"
server_key_iv: "10745b8e33617240c339850dfca7b537"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.742105 13877 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 21:59:10.742928 13892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.743117 13877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:10.743194 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal
uuid: "9543330b81d8478990d6e37386518b81"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "5e46bd1ef08055726749f611a80c8cce"
server_key_iv: "10745b8e33617240c339850dfca7b537"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.743283 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.756202 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:10.758945 13877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:10.759122 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:10.763746 13877 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.188:46449
I20260620 21:59:10.763813 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.188:46449 every 8 connection(s)
I20260620 21:59:10.764246 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
I20260620 21:59:10.765321 13945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:10.767047 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.188@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:10.774114 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 13877
I20260620 21:59:10.776822 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.767914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.6.241.190:43375)
0620 21:59:10.767984 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.767986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.768134 (+ 148us) server_negotiation.cc:407] Connection header received
0620 21:59:10.768659 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.768660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.768672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.768731 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.769405 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.769914 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.771025 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.771230 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.771318 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.773536 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.773538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.773539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.773547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.775295 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.775801 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.775801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.775802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.775853 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.776123 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.776124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.776125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.776283 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.776355 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.776647 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.776728 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:59:10.777033 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.767770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:59978)
0620 21:59:10.767993 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.768040 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.768538 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.768827 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.768830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.769104 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.769307 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.769311 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.770021 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.770022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.770936 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.770938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.771341 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.771342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.771466 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.771993 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.772002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.773432 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.775366 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.775373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.775376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.775725 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.775925 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.775926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.775927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.776048 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.776356 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.776363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.776506 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.776608 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.776788 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":111,"threads_started":1}
I20260620 21:59:10.777631 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.241.188@KRBTEST.COM
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.188@KRBTEST.COM for oryx/127.6.241.189@KRBTEST.COM
I20260620 21:59:10.783805 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.774813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.6.241.188:46449)
0620 21:59:10.775010 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.775015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.775033 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:59:10.775052 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.775054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.775066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.775148 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.775858 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.776582 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.777543 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.777987 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.778055 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.780684 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.780693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.780693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.780700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.782557 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.782957 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.782960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.782961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.783021 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.783221 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.783222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.783222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.783377 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.783521 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.783608 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.783682 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
I20260620 21:59:10.785122 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.777887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.0.0.1:52046)
0620 21:59:10.777938 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.777950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.777996 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.778125 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.778126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.778293 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.778357 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.778359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.779428 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.779429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.780078 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.780080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.780532 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.780533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.780605 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.781038 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.781043 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.782516 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.784348 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.784349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.784349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.784561 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.784705 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.784706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.784706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.784748 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.784916 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.784916 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.784947 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.784995 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.785063 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 21:59:10.785140 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.777976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.6.241.189:32981)
0620 21:59:10.778020 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.778023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.778033 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:10.778061 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.778062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.778075 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.778125 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.778468 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.779312 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.780166 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.780451 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.780539 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.782634 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.782635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.782636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.782643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.784262 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.784611 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.784611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.784612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.784661 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.784781 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.784781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.784782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.784869 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.784912 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.785010 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.785077 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 21:59:10.785530 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.188@KRBTEST.COM for oryx/127.6.241.188@KRBTEST.COM
I20260620 21:59:10.791563 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.785780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:46449 (local address 127.0.0.1:46696)
0620 21:59:10.785827 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.785838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.785877 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.785974 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.785975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.786080 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.786110 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.786111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.786780 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.786781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.787260 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.787261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.787671 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.787672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.787709 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.788014 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.788018 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.789299 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.790701 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.790702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.790703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.790874 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.791043 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.791044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.791044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.791094 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.791314 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.791315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.791348 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.791397 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.791472 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 21:59:10.791610 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.785844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.6.241.188:46449)
0620 21:59:10.785873 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.785874 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.785881 (+ 7us) server_negotiation.cc:407] Connection header received
0620 21:59:10.785929 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.785930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.785937 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.785973 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.786187 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.786720 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.787394 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.787599 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.787668 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.789393 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.789395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.789395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.789401 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.790557 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.790941 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.790942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.790942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.790989 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.791170 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.791171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.791171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.791265 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.791313 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.791411 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.791517 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":15}
I20260620 21:59:10.792989 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Bootstrap starting.
I20260620 21:59:10.793812 13945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:10.794085 13945 log.cc:824] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:10.794869 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: No bootstrap required, opened a new log
I20260620 21:59:10.796442 13945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } }
I20260620 21:59:10.796564 13945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:10.796586 13945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9543330b81d8478990d6e37386518b81, State: Initialized, Role: FOLLOWER
I20260620 21:59:10.796746 13945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } }
I20260620 21:59:10.797012 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } } }
I20260620 21:59:10.797079 13945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:10.797104 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 21:59:10.798856 13961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:10.798918 13961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:10.800981 13877 master_runner.cc:428] Detected that this master 9543330b81d8478990d6e37386518b81 is joining an existing cluster
I20260620 21:59:10.801035 13877 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.241.188:46449
I20260620 21:59:10.809603 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.802056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.6.241.189:32981)
0620 21:59:10.802098 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.802100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.802662 (+ 562us) server_negotiation.cc:407] Connection header received
0620 21:59:10.802802 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.802803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.802814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.802865 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.803750 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.804482 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.805433 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.805748 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.805819 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.807116 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.807117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.807117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.807123 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.808553 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.808945 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.808946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.808946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.808993 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.809177 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.809177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.809178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.809274 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.809320 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.809476 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.809538 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 21:59:10.809883 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.802033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.0.0.1:52062)
0620 21:59:10.802568 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.802579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.802678 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.802946 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.802947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.803176 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.803259 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.803261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.804561 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.804562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.805355 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.805357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.805886 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.805888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.805957 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.806574 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.806579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.807033 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.808634 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.808635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.808635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.808875 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.809052 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.809053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.809054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.809123 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.809328 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.809329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.809378 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.809461 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.809566 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":454,"threads_started":1}
I20260620 21:59:10.815713 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.802960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.6.241.188:46449)
0620 21:59:10.802995 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.802998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.810127 (+ 7129us) server_negotiation.cc:407] Connection header received
0620 21:59:10.810190 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.810191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.810199 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.810238 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.810516 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.811061 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.812048 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.812214 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.812261 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.813238 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.813240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.813240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.813246 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.814679 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.815027 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.815028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.815028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.815076 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.815250 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.815250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.815250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.815380 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.815431 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.815567 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.815648 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 21:59:10.815830 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.803027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:46449 (local address 127.0.0.1:46702)
0620 21:59:10.810067 (+ 7040us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.810077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.810122 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.810244 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.810245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.810399 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.810454 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.810456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.811131 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.811132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.811974 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.811976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.812277 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.812278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.812345 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.812780 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.812785 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.813152 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.814749 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.814750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.814750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.814959 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.815131 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.815132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.815132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.815195 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.815437 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.815438 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.815480 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.815551 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.815712 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7028,"mutex_wait_us":55,"thread_start_us":58,"threads_started":1}
I20260620 21:59:10.830950 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.802730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.6.241.190:43375)
0620 21:59:10.802775 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.802777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.802953 (+ 176us) server_negotiation.cc:407] Connection header received
0620 21:59:10.803062 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.803063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.803074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.803104 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.803486 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.804071 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.805181 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.805662 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.805737 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.828368 (+ 22631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.828371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.828371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.828381 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.829741 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.830109 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.830110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.830111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.830180 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.830370 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.830371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.830371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.830494 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.830569 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.830757 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.830846 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:59:10.830964 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.802748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:59990)
0620 21:59:10.802897 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.802907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.802976 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.803203 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.803204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.803370 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.803429 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.803430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.804155 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.804156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.805073 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.805076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.805891 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.805892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.805996 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.827295 (+ 21299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.827302 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.828237 (+ 935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.829837 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.829839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.829839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.830023 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.830245 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.830246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.830246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.830303 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.830559 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.830561 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.830627 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.830730 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.830848 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260620 21:59:10.835163 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 13287
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992749, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.190@KRBTEST.COM for oryx/127.6.241.188@KRBTEST.COM
I20260620 21:59:10.840768 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.833174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.6.241.188:46449)
0620 21:59:10.833228 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.833230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.833239 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:10.833274 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.833275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.833285 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.833354 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.833610 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.834341 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.835670 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.835947 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.836045 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.837824 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.837825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.837826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.837833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.839639 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.840000 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.840002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.840002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.840059 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.840245 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.840246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.840247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.840364 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.840433 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.840587 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.840667 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 21:59:10.840770 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.833097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:46449 (local address 127.0.0.1:46714)
0620 21:59:10.833151 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.833163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.833205 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.833354 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.833355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.833486 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.833530 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.833531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.834427 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.834428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.835260 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.835263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.836045 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.836046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.836114 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.836547 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.836553 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.837738 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.839740 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.839741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.839742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.839917 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.840121 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.840122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.840123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.840180 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.840435 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.840436 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.840483 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.840563 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.840676 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 21:59:10.841645 13225 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.241.188:46449: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9543330b81d8478990d6e37386518b81"
member_type: NON_VOTER
last_known_addr {
host: "127.6.241.188"
port: 46449
}
attrs {
promote: true
}
}
dest_uuid: "782f2eef2b6e43c3a1f1122b1598d935"
cas_config_opid_index: 5
I20260620 21:59:10.841917 13225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } }
I20260620 21:59:10.842470 13785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9543330b81d8478990d6e37386518b81 (127.6.241.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.842882 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:44265
--local_ip_for_outbound_sockets=127.6.241.129
--tserver_master_addrs=127.6.241.190:43375
--webserver_port=35585
--webserver_interface=127.6.241.129
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab
--principal=oryx/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.241.190:43375,127.6.241.189:32981
--tserver_master_addrs=127.6.241.190:43375,127.6.241.189:32981,127.6.241.188:46449 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:59:10.843623 13785 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.241.188:46449
I20260620 21:59:10.843706 13785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.843786 13785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:10.844205 13919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:10.845062 13919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Refusing update from remote peer 782f2eef2b6e43c3a1f1122b1598d935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 21:59:10.845372 13877 master.cc:537] Master@127.6.241.188:46449 shutting down...
I20260620 21:59:10.845428 13784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:59:10.846552 13212 consensus_peers.cc:487] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 -> Peer 9543330b81d8478990d6e37386518b81 (127.6.241.188:46449): Couldn't send request to peer 9543330b81d8478990d6e37386518b81. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:59:10.848886 13877 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:10.848990 13877 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:10.849045 13877 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: stopping tablet replica
I20260620 21:59:10.849156 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.842373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.0.0.1:52068)
0620 21:59:10.842745 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.842756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.842796 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.843014 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.843015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.843016 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:10.843057 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.843059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.843983 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.843984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.845279 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.845282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.848785 (+ 3503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.848787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.848947 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.848979 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.849031 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.849080 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354}
I20260620 21:59:10.849402 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.842424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.6.241.189:32981)
0620 21:59:10.842465 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.842467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.842853 (+ 386us) server_negotiation.cc:407] Connection header received
0620 21:59:10.842882 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.842883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.842905 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.842961 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:10.843127 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.843886 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.846960 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.848566 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.848631 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.848716 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.849291 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.849337 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 21:59:10.849606 13833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 LEARNER]: Refusing update from remote peer 782f2eef2b6e43c3a1f1122b1598d935: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 21:59:10.849789 13784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 21:59:10.850333 13833 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9543330b81d8478990d6e37386518b81 (127.6.241.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.851580 13862 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782f2eef2b6e43c3a1f1122b1598d935. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.851668 13862 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: This master's current role is: LEARNER
I20260620 21:59:10.851701 13862 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: NON_VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: true } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.851747 13862 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: This master's current role is: LEARNER
I20260620 21:59:10.852154 13784 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: attempting to promote NON_VOTER d4d0aab8261446af860d68672e92d0de to VOTER
I20260620 21:59:10.852371 13784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } }
I20260620 21:59:10.852977 13833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 LEARNER]: Refusing update from remote peer 782f2eef2b6e43c3a1f1122b1598d935: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 21:59:10.853147 13785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 21:59:10.853472 13212 consensus_peers.cc:487] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 -> Peer 9543330b81d8478990d6e37386518b81 (127.6.241.188:46449): Couldn't send request to peer 9543330b81d8478990d6e37386518b81. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:59:10.854400 13785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d4d0aab8261446af860d68672e92d0de (127.6.241.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.855340 13785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.855414 13785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:10.855465 13785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.855512 13785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:10.855751 13833 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d4d0aab8261446af860d68672e92d0de (127.6.241.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.856712 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.856794 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:10.865312 13877 master.cc:552] Master@127.6.241.188:46449 shutdown complete.
I20260620 21:59:10.865381 13877 master_runner.cc:305] Clearing existing system tablet
I20260620 21:59:10.865515 13877 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:10.865566 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:10.866339 13877 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:10.866875 13975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.867017 13877 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.867079 13877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.867102 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal
uuid: "9543330b81d8478990d6e37386518b81"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "5e46bd1ef08055726749f611a80c8cce"
server_key_iv: "10745b8e33617240c339850dfca7b537"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.867813 13877 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:10.869340 13877 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 21:59:10.869414 13877 log.cc:1196] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 21:59:10.869611 13877 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Deleting consensus metadata
I20260620 21:59:10.869725 13877 master_runner.cc:315] Copying system tablet from 127.6.241.190:43375
I20260620 21:59:10.870337 13877 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: tablet copy: Beginning tablet copy session from remote peer at address 127.6.241.190:43375
I20260620 21:59:10.877767 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.870854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.6.241.190:43375)
0620 21:59:10.870898 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.870900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.870939 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:59:10.870967 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.870969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.870979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.871037 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.871418 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.872194 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.873304 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.873595 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.873661 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.874871 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.874873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.874873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.874881 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:10.876504 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.876915 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.876916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.876917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.876981 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:10.877189 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:10.877190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:10.877190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:10.877328 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:10.877394 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:10.877587 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 21:59:10.877669 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 21:59:10.877782 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.870695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:59998)
0620 21:59:10.870817 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.870828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.870879 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.871050 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.871051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.871252 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.871323 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.871326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.872288 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.872290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.873182 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.873187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.873672 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.873674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.873740 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.874296 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.874301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.874781 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:10.876604 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.876606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.876606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.876831 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.877047 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:10.877048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:10.877048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:10.877120 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:10.877393 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:10.877394 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:10.877443 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 21:59:10.877554 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 21:59:10.877675 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260620 21:59:10.878444 13245 tablet_copy_service.cc:140] P 782f2eef2b6e43c3a1f1122b1598d935: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9543330b81d8478990d6e37386518b81 ({username='oryx', principal='oryx/127.6.241.188@KRBTEST.COM'} at 127.0.0.1:59998)
I20260620 21:59:10.878513 13245 tablet_copy_service.cc:161] P 782f2eef2b6e43c3a1f1122b1598d935: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9543330b81d8478990d6e37386518b81 at {username='oryx', principal='oryx/127.6.241.188@KRBTEST.COM'} at 127.0.0.1:59998: session id = 9543330b81d8478990d6e37386518b81-00000000000000000000000000000000
I20260620 21:59:10.879267 13245 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 21:59:10.879742 13877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 21:59:10.881078 13877 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: tablet copy: Starting download of 0 data blocks...
I20260620 21:59:10.881229 13877 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: tablet copy: Starting download of 1 WAL segments...
I20260620 21:59:10.882050 13877 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 21:59:10.882995 13245 tablet_copy_service.cc:342] P 782f2eef2b6e43c3a1f1122b1598d935: Request end of tablet copy session 9543330b81d8478990d6e37386518b81-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.241.188@KRBTEST.COM'} at 127.0.0.1:59998
I20260620 21:59:10.883054 13245 tablet_copy_service.cc:433] P 782f2eef2b6e43c3a1f1122b1598d935: ending tablet copy session 9543330b81d8478990d6e37386518b81-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9543330b81d8478990d6e37386518b81
I20260620 21:59:10.885787 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:10.887009 13982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.887066 13985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.887231 13877 server_base.cc:1034] running on GCE node
W20260620 21:59:10.887645 13983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.887840 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:10.888005 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:10.889139 13877 hybrid_clock.cc:648] HybridClock initialized: now 1781992750889127 us; error 33 us; skew 500 ppm
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:10.890782 13877 init.cc:377] Logged in from keytab as oryx/127.6.241.188@KRBTEST.COM (short username oryx)
I20260620 21:59:10.891182 13877 webserver.cc:460] Webserver started at http://127.6.241.188:39751/ using document root <none> and password file <none>
I20260620 21:59:10.891311 13877 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:10.891350 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:10.892181 13877 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:10.892802 13992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.892972 13877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.893031 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal
uuid: "9543330b81d8478990d6e37386518b81"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "5e46bd1ef08055726749f611a80c8cce"
server_key_iv: "10745b8e33617240c339850dfca7b537"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.893101 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.900275 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:10.902485 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:10.905758 13877 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.188:46449
I20260620 21:59:10.906132 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
I20260620 21:59:10.906700 14044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.188:46449 every 8 connection(s)
I20260620 21:59:10.907192 14045 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:59:10.907732 14045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Bootstrap starting.
I20260620 21:59:10.911054 14045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:59:10.911311 14045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Bootstrap complete.
I20260620 21:59:10.911547 14045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } }
I20260620 21:59:10.911664 14045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9543330b81d8478990d6e37386518b81, State: Initialized, Role: LEARNER
I20260620 21:59:10.911707 14045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } }
I20260620 21:59:10.911902 14045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:10.911916 14048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:10.912134 14048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: This master's current role is: LEARNER
I20260620 21:59:10.913380 14059 catalog_manager.cc:1259] Loaded cluster ID: 64f670dccdbc4b8cbbf599ac69360f5d
I20260620 21:59:10.913465 14059 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: loading cluster ID for follower catalog manager: success
I20260620 21:59:10.914675 14059 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: acquiring CA information for follower catalog manager: success
I20260620 21:59:10.915024 14059 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 21:59:10.944687 13970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:10.944936 13970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:10.944979 13970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:10.946213 13970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:10.946290 13970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:10.946338 13970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:10.947942 13970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab
--principal=oryx/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:44265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=35585
--tserver_master_addrs=127.6.241.190:43375,127.6.241.189:32981,127.6.241.188:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:10.948243 13970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:10.948480 13970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:10.954453 14065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.954479 14067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:10.954447 14064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:10.954780 13970 server_base.cc:1034] running on GCE node
I20260620 21:59:10.954932 13970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:10.955148 13970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:10.956315 13970 hybrid_clock.cc:648] HybridClock initialized: now 1781992750956287 us; error 44 us; skew 500 ppm
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:10.958890 13970 init.cc:377] Logged in from keytab as oryx/127.6.241.129@KRBTEST.COM (short username oryx)
I20260620 21:59:10.959409 13970 webserver.cc:460] Webserver started at http://127.6.241.129:35585/ using document root <none> and password file <none>
I20260620 21:59:10.959661 13970 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:10.959736 13970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:10.961136 13970 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:10.961750 14074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.961895 13970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:10.961982 13970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "da9da6d6ed39410d914401852c1b60a5"
format_stamp: "Formatted at 2026-06-20 21:59:09 on dist-test-slave-gl2g"
server_key: "55b6daab21c60f4d9964183f35a19db3"
server_key_iv: "ed0ff48e99f3a5e700441a650c3ba0f2"
server_key_version: "encryptionkey@0"
I20260620 21:59:10.962347 13970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:10.980929 13970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:10.983484 13970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:10.983639 13970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:10.983879 13970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:10.984213 13970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:10.984264 13970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.984309 13970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:10.984335 13970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:10.991658 13970 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:44265
I20260620 21:59:10.991695 14187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:44265 every 8 connection(s)
I20260620 21:59:10.992064 13970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.188@KRBTEST.COM
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
Jun 20 21:59:10 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992750, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.189@KRBTEST.COM
I20260620 21:59:11.000563 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 13970
I20260620 21:59:11.000801 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 13421
I20260620 21:59:11.004272 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.993134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:48423 (local address 127.6.241.189:32981)
0620 21:59:10.993253 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.993255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.993265 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:10.993770 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.993771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.993783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.993839 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.994861 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.995607 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.996647 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.996945 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.996985 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.000383 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.000384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.000385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.000393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.002653 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.003220 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.003221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.003222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.003283 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.003593 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.003594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.003594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.003732 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.003768 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.004086 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.004169 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 21:59:11.004689 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.992842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.6.241.129:48423)
0620 21:59:10.993017 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.993057 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.993669 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.993918 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.993922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.994437 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.994751 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.994757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.995700 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.995701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.996556 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.996558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.997058 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.997058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.997133 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.997810 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.997814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:11.000294 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:11.002752 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.002761 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.002764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.003111 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.003363 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.003364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.003364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.003516 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.003806 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:11.003811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:11.003939 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.004068 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.004256 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":6,"spinlock_wait_cycles":7936,"thread_start_us":91,"threads_started":1}
I20260620 21:59:11.005252 14190 heartbeater.cc:347] Connected to a master server at 127.6.241.189:32981
I20260620 21:59:11.005366 14190 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:11.005558 14190 heartbeater.cc:511] Master 127.6.241.189:32981 requested a full tablet report, sending...
I20260620 21:59:11.006145 13823 ts_manager.cc:194] Registered new tserver with Master: da9da6d6ed39410d914401852c1b60a5 (127.6.241.129:44265)
I20260620 21:59:11.006824 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:44769
--local_ip_for_outbound_sockets=127.6.241.130
--tserver_master_addrs=127.6.241.190:43375
--webserver_port=40757
--webserver_interface=127.6.241.130
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab
--principal=oryx/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.241.190:43375,127.6.241.189:32981
--tserver_master_addrs=127.6.241.190:43375,127.6.241.189:32981,127.6.241.188:46449 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:59:11.007951 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.993322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:44879 (local address 127.6.241.190:43375)
0620 21:59:10.993388 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.993390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.993452 (+ 62us) server_negotiation.cc:407] Connection header received
0620 21:59:10.993821 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.993823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.993837 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.993871 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.994842 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.995429 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.996633 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.996869 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.996935 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.000447 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.000449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.000449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.000457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.003544 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.007027 (+ 3483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.007029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.007030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.007107 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.007299 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.007300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.007300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.007469 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.007542 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.007773 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.007855 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:59:11.007966 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.993147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.129:44879)
0620 21:59:10.993402 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.993411 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.993702 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.994316 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.994317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.994491 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.994750 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.994756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.995529 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.995530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.996533 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.996536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.996953 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.996954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.997085 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.997766 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.997778 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.999923 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:11.006660 (+ 6737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.006661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.006661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.006918 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.007178 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.007178 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.007178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.007237 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.007578 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:11.007579 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:11.007641 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.007732 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.007861 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260620 21:59:11.008574 14189 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43375
I20260620 21:59:11.008633 14189 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:11.008716 14189 heartbeater.cc:511] Master 127.6.241.190:43375 requested a full tablet report, sending...
I20260620 21:59:11.008981 13225 ts_manager.cc:194] Re-registered known tserver with Master: da9da6d6ed39410d914401852c1b60a5 (127.6.241.129:44265)
I20260620 21:59:11.009505 13225 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:44879
I20260620 21:59:11.009588 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.992808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:34899 (local address 127.6.241.188:46449)
0620 21:59:10.993218 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:10.993221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:10.993403 (+ 182us) server_negotiation.cc:407] Connection header received
0620 21:59:10.993839 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:10.993840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:10.993854 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:10.993918 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:10.994860 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.995950 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.997005 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.997330 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.997397 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.999682 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:10.999733 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:10.999733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:10.999741 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.008329 (+ 8588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.008732 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.008734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.008734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.008799 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.008998 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.008999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.008999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.009159 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.009233 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.009377 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.009464 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":58,"threads_started":1}
I20260620 21:59:11.009833 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.992842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:46449 (local address 127.6.241.129:34899)
0620 21:59:10.993324 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.993333 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:10.993705 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:10.993984 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:10.993985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:10.994458 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:10.994751 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.994757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.996053 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.996054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.996923 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:10.996925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:10.997403 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:10.997403 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:10.997463 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:10.997955 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:10.997960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:10.999564 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:11.008416 (+ 8852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.008417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.008418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.008639 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.008879 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.008879 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.008880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.008934 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.009225 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:11.009226 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:11.009268 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.009352 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.009463 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":65,"thread_start_us":91,"threads_started":1}
I20260620 21:59:11.010234 14188 heartbeater.cc:347] Connected to a master server at 127.6.241.188:46449
I20260620 21:59:11.010299 14188 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:11.010382 14188 heartbeater.cc:511] Master 127.6.241.188:46449 requested a full tablet report, sending...
I20260620 21:59:11.010799 14009 ts_manager.cc:194] Registered new tserver with Master: da9da6d6ed39410d914401852c1b60a5 (127.6.241.129:44265)
W20260620 21:59:11.093881 14196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:11.094105 14196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:11.094131 14196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:11.095480 14196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:11.095527 14196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:11.095580 14196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:11.097376 14196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab
--principal=oryx/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:44769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=40757
--tserver_master_addrs=127.6.241.190:43375,127.6.241.189:32981,127.6.241.188:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:11.097648 14196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:11.097899 14196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:11.103171 14202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:11.103192 14201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:11.103171 14204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:11.103524 14196 server_base.cc:1034] running on GCE node
I20260620 21:59:11.103751 14196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:11.104017 14196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:11.105180 14196 hybrid_clock.cc:648] HybridClock initialized: now 1781992751105169 us; error 35 us; skew 500 ppm
Jun 20 21:59:11 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:11.107908 14196 init.cc:377] Logged in from keytab as oryx/127.6.241.130@KRBTEST.COM (short username oryx)
I20260620 21:59:11.108332 14196 webserver.cc:460] Webserver started at http://127.6.241.130:40757/ using document root <none> and password file <none>
I20260620 21:59:11.108522 14196 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:11.108577 14196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:11.109910 14196 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:11.110469 14211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:11.110702 14196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:11.110795 14196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "93e1b09488874feab266f10e487666cc"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "c50ae6a5bc686e9d5da7a241c58c034a"
server_key_iv: "e84740a2706b1217d108803390102893"
server_key_version: "encryptionkey@0"
I20260620 21:59:11.111151 14196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:11.132301 14196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:11.134761 14196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:11.134895 14196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:11.135085 14196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:11.135437 14196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:11.135488 14196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:11.135529 14196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:11.135558 14196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:11.141358 14196 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:44769
I20260620 21:59:11.141429 14324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:44769 every 8 connection(s)
I20260620 21:59:11.141786 14196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:11.143683 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 14196
I20260620 21:59:11.143854 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 13555
I20260620 21:59:11.150315 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:41127
--local_ip_for_outbound_sockets=127.6.241.131
--tserver_master_addrs=127.6.241.190:43375
--webserver_port=43295
--webserver_interface=127.6.241.131
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab
--principal=oryx/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.241.190:43375,127.6.241.189:32981
--tserver_master_addrs=127.6.241.190:43375,127.6.241.189:32981,127.6.241.188:46449 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:59:11 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
Jun 20 21:59:11 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.188@KRBTEST.COM
Jun 20 21:59:11 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.189@KRBTEST.COM
I20260620 21:59:11.156082 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.142564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:52345 (local address 127.6.241.189:32981)
0620 21:59:11.142639 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.142642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.142934 (+ 292us) server_negotiation.cc:407] Connection header received
0620 21:59:11.143617 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.143618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.143627 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.143650 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.147542 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.148255 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.149297 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.149609 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.149652 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.152355 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.152357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.152357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.152366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.154253 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.154877 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.154878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.154879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.154939 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.155254 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.155255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.155255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.155411 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.155466 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.155878 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.155970 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:59:11.156163 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.142978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:54541 (local address 127.6.241.188:46449)
0620 21:59:11.143021 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.143022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.143031 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:11.143694 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.143695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.143706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.143764 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.145095 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.145905 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.147561 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.147946 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.147991 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.152355 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.152357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.152357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.152366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.154101 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.154881 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.154882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.154882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.154950 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.155257 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.155258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.155258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.155416 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.155477 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.155889 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.156001 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 21:59:11.156571 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.142527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:40247 (local address 127.6.241.190:43375)
0620 21:59:11.142660 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.142662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.142887 (+ 225us) server_negotiation.cc:407] Connection header received
0620 21:59:11.143501 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.143503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.143516 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.143568 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.145247 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.145964 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.147541 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.147841 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.147884 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.152415 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.152417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.152418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.152426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.155272 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.155691 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.155693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.155693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.155756 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.155951 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.155952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.155952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.156103 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.156157 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.156350 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.156484 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 21:59:11.156685 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.142548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.6.241.130:52345)
0620 21:59:11.142735 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.142778 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.143385 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.147193 (+ 3808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.147194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.147376 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:11.147435 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.147437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.148361 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.148361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.149196 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.149199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.149734 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.149735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.149807 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.150240 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:11.150245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:11.152239 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:11.154348 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.154349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.154350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.154784 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.155012 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.155013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.155013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.155159 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.155498 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:11.155505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:11.155694 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.156238 (+ 544us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.156437 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":118,"threads_started":1}
I20260620 21:59:11.156904 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.142548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.130:40247)
0620 21:59:11.142831 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.142839 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.143385 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.144866 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.144867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.145078 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:11.145150 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.145153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.146051 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.146052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.147448 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.147450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.147976 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.147976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.148122 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.148901 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:11.148906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:11.152286 (+ 3380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:11.155392 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.155392 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.155393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.155595 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.155827 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.155828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.155828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.155883 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.156193 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:11.156194 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:11.156237 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.156684 (+ 447us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.156795 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":85,"thread_start_us":121,"threads_started":1}
I20260620 21:59:11.157096 14327 heartbeater.cc:347] Connected to a master server at 127.6.241.189:32981
I20260620 21:59:11.157199 14327 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:11.157277 14326 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43375
I20260620 21:59:11.157323 14326 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:11.157382 14327 heartbeater.cc:511] Master 127.6.241.189:32981 requested a full tablet report, sending...
I20260620 21:59:11.157392 14326 heartbeater.cc:511] Master 127.6.241.190:43375 requested a full tablet report, sending...
I20260620 21:59:11.157812 13225 ts_manager.cc:194] Re-registered known tserver with Master: 93e1b09488874feab266f10e487666cc (127.6.241.130:44769)
I20260620 21:59:11.157845 13823 ts_manager.cc:194] Registered new tserver with Master: 93e1b09488874feab266f10e487666cc (127.6.241.130:44769)
I20260620 21:59:11.157912 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.142890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:46449 (local address 127.6.241.130:54541)
0620 21:59:11.142990 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.142998 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.143394 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.143905 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.143911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.144789 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:11.144993 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.144998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.146025 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.146026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.147453 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.147456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.148067 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.148068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.148141 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.148803 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:11.148813 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:11.152232 (+ 3419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:11.154217 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.154225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.154228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.154777 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.155031 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.155032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.155032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.155163 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.155512 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:11.155513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:11.155695 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.157395 (+ 1700us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.157601 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":34,"thread_start_us":58,"threads_started":1}
I20260620 21:59:11.158310 13225 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:40247
I20260620 21:59:11.158354 14325 heartbeater.cc:347] Connected to a master server at 127.6.241.188:46449
I20260620 21:59:11.158392 14325 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:11.158468 14325 heartbeater.cc:511] Master 127.6.241.188:46449 requested a full tablet report, sending...
I20260620 21:59:11.158814 14009 ts_manager.cc:194] Registered new tserver with Master: 93e1b09488874feab266f10e487666cc (127.6.241.130:44769)
I20260620 21:59:11.236908 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.234022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.6.241.188:46449)
0620 21:59:11.234094 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.234096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.234106 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:11.234135 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.234136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.234197 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.234268 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:11.234406 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.235029 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.236157 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.236570 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.236657 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.236730 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.236793 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.236827 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:59:11.236908 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.233930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:46449 (local address 127.0.0.1:46722)
0620 21:59:11.233992 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.234005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.234075 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.234272 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.234273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.234275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:11.234331 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.234333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.235124 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.235125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.236069 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.236072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.236658 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.236659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.236724 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.236727 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.236782 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.236830 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 21:59:11.237229 14019 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 21:59:11.237386 14048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782f2eef2b6e43c3a1f1122b1598d935. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:11.237500 14048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: This master's current role is: LEARNER
W20260620 21:59:11.237973 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:11.238199 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:11.238242 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:11.239523 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:11.239593 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:11.239642 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:11.241228 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab
--principal=oryx/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:41127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=43295
--tserver_master_addrs=127.6.241.190:43375,127.6.241.189:32981,127.6.241.188:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:11.241526 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:11.241778 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:11.246902 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:11.246922 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:11.246922 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:11.247067 14332 server_base.cc:1034] running on GCE node
I20260620 21:59:11.247278 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:11.247576 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:11.248749 14332 hybrid_clock.cc:648] HybridClock initialized: now 1781992751248736 us; error 45 us; skew 500 ppm
Jun 20 21:59:11 dist-test-slave-gl2g krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:11.251437 14332 init.cc:377] Logged in from keytab as oryx/127.6.241.131@KRBTEST.COM (short username oryx)
I20260620 21:59:11.251950 14332 webserver.cc:460] Webserver started at http://127.6.241.131:43295/ using document root <none> and password file <none>
I20260620 21:59:11.252192 14332 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:11.252270 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:11.253690 14332 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 21:59:11.254326 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:11.254511 14332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:11.254647 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "60e8765db7344cf180a4df768e8623be"
format_stamp: "Formatted at 2026-06-20 21:59:10 on dist-test-slave-gl2g"
server_key: "6198210a6ea17597b4e83fec2f3a0a4d"
server_key_iv: "f9db6a341b455092930d0e6db217a104"
server_key_version: "encryptionkey@0"
I20260620 21:59:11.255050 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:11.269709 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:11.271901 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:11.272079 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:11.272341 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:11.272686 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:11.272746 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:11.272789 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:11.272826 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:11.279378 14332 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:41127
I20260620 21:59:11.279420 14460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:41127 every 8 connection(s)
I20260620 21:59:11.279875 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:11.286185 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 14332
Jun 20 21:59:11 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.188@KRBTEST.COM
Jun 20 21:59:11 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.189@KRBTEST.COM
Jun 20 21:59:11 dist-test-slave-gl2g krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:11.292341 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.281411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:41775 (local address 127.6.241.188:46449)
0620 21:59:11.281562 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.281564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.281577 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:59:11.282452 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.282454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.282473 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.282510 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.283346 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.284117 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.285086 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.285340 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.285407 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.288578 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.288580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.288581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.288590 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.290376 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.291000 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.291002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.291002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.291064 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.291407 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.291408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.291409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.291546 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.291684 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.292139 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.292228 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 21:59:11.292966 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.281209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:46449 (local address 127.6.241.131:41775)
0620 21:59:11.281316 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.281325 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.282320 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.282600 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.282627 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.283082 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:11.283226 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.283233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.284216 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.284217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.284971 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.284972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.285420 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.285421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.285569 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.286499 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:11.286503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:11.288453 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:11.290493 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.290505 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.290508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.290897 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.291141 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.291142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.291142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.291327 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.291722 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:11.291730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:11.291926 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.292456 (+ 530us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.292677 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260620 21:59:11.293550 14461 heartbeater.cc:347] Connected to a master server at 127.6.241.188:46449
I20260620 21:59:11.293677 14461 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:11.293929 14461 heartbeater.cc:511] Master 127.6.241.188:46449 requested a full tablet report, sending...
I20260620 21:59:11.294412 14009 ts_manager.cc:194] Registered new tserver with Master: 60e8765db7344cf180a4df768e8623be (127.6.241.131:41127)
I20260620 21:59:11.294999 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.280838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.6.241.131:36177)
0620 21:59:11.281124 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.281139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.282317 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.282636 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.282636 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.283068 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:11.283226 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.283233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.284410 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.284410 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.285197 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.285199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.285618 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.285619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.285688 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.286707 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:11.286711 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:11.289203 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:11.291365 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.291366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.291367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.291637 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.293900 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.293901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.293901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.293976 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.294579 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:11.294580 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:11.294650 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.294734 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.294858 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260620 21:59:11.295389 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.281488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:36177 (local address 127.6.241.189:32981)
0620 21:59:11.281579 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.281581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.281591 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:11.282452 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.282454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.282467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.282508 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.283346 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.284117 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.285270 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.285533 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.285605 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.289308 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.289310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.289311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.289320 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.291242 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.293696 (+ 2454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.293697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.293697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.293772 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.294062 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.294063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.294063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.294465 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.294537 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.295203 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.295290 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:59:11.296087 14463 heartbeater.cc:347] Connected to a master server at 127.6.241.189:32981
I20260620 21:59:11.296140 14463 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:11.296255 14463 heartbeater.cc:511] Master 127.6.241.189:32981 requested a full tablet report, sending...
I20260620 21:59:11.296602 13823 ts_manager.cc:194] Registered new tserver with Master: 60e8765db7344cf180a4df768e8623be (127.6.241.131:41127)
I20260620 21:59:11.297798 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.280829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:36009 (local address 127.6.241.190:43375)
0620 21:59:11.281328 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.281332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.281345 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:59:11.282434 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.282436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.282452 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.282502 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.283338 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.283850 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.285246 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.285527 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.285569 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.291737 (+ 6168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.291739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.291740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.291749 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.293587 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.296044 (+ 2457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.296045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.296046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.296111 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.296941 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.296942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.296942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.297086 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.297154 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.297607 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.297695 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471}
I20260620 21:59:11.298043 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.280838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:36009)
0620 21:59:11.281073 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.281128 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.282285 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.282589 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.282620 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.282988 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:11.283226 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.283231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.283966 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.283967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.285128 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.285130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.285648 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.285648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.285720 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.286465 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:11.286477 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:11.289354 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:11.295686 (+ 6332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.295687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.295688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.295949 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.296792 (+ 843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:11.296793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:11.296793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:11.296859 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:11.297222 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:11.297223 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:11.297292 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.297351 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.297461 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260620 21:59:11.298369 14462 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43375
I20260620 21:59:11.298421 14462 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:11.298491 14462 heartbeater.cc:511] Master 127.6.241.190:43375 requested a full tablet report, sending...
I20260620 21:59:11.298794 13225 ts_manager.cc:194] Re-registered known tserver with Master: 60e8765db7344cf180a4df768e8623be (127.6.241.131:41127)
I20260620 21:59:11.299182 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.289806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46728 (local address 127.6.241.188:46449)
0620 21:59:11.290038 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.290040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.290287 (+ 247us) server_negotiation.cc:407] Connection header received
0620 21:59:11.290381 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.290382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.290396 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.290440 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.291265 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.292016 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.293355 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.293717 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.293778 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.295342 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.295344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.295344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.295350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.297641 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.298031 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.298032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.298032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.298106 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.298341 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.298342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.298343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.298495 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.298624 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.298987 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.299087 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":57,"threads_started":1}
I20260620 21:59:11.299293 13225 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:36009
I20260620 21:59:11.301575 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.287764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.6.241.189:32981)
0620 21:59:11.287890 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.287892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.289806 (+ 1914us) server_negotiation.cc:407] Connection header received
0620 21:59:11.289833 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.289834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.289850 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.289908 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.290830 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.291668 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.294209 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.295378 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.295419 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.297365 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.297366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.297367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.297374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.300213 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.300585 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.300587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.300587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.300657 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.300890 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.300890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.300891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.300998 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.301125 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.301257 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.301361 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260620 21:59:11.301972 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.289514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.6.241.190:43375)
0620 21:59:11.293429 (+ 3915us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.293431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.293440 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:11.293464 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.293465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.293478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.294354 (+ 876us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.294401 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.295242 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.296388 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.296738 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.297053 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.298278 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.298280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.298280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.298287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.300674 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.301064 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.301065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.301066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.301123 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.301351 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.301352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.301352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.301515 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.301594 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.301764 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.301854 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3875,"thread_start_us":116,"threads_started":1}
I20260620 21:59:11.303735 13225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:11.305473 13225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:11.306367 13784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: NON_VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: true } } }
I20260620 21:59:11.306478 13784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:11.307909 14479 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: attempting to promote NON_VOTER 9543330b81d8478990d6e37386518b81 to VOTER
I20260620 21:59:11.308080 14479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } }
I20260620 21:59:11.308569 13833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Refusing update from remote peer 782f2eef2b6e43c3a1f1122b1598d935: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 21:59:11.308796 13973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 21:59:11.309140 14018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 LEARNER]: Refusing update from remote peer 782f2eef2b6e43c3a1f1122b1598d935: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 21:59:11.309381 13973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 21:59:11.309508 13973 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9543330b81d8478990d6e37386518b81 (127.6.241.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:11.309700 13833 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9543330b81d8478990d6e37386518b81 (127.6.241.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:11.309809 14019 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9543330b81d8478990d6e37386518b81 (127.6.241.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:11.310631 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:11.310724 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:11.310715 14479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:11.310786 14479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:11.310933 14048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:11.311020 14048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:11.311100 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:11.311193 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:11.311437 13784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "782f2eef2b6e43c3a1f1122b1598d935" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:11.311568 13784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:11.317092 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.312512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:44769 (local address 127.0.0.1:59804)
0620 21:59:11.312601 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.312616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.312663 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.313487 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.313488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.313489 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:11.313521 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.313522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.314582 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.314582 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.315824 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.315827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.316866 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.316867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.316928 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.316932 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.316964 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.317009 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":6}
I20260620 21:59:11.317096 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.312512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:44265 (local address 127.0.0.1:35518)
0620 21:59:11.312700 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.312711 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.312789 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.313371 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.313373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.313375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:11.313445 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.313449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.314446 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.314448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.315825 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.315827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.316847 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.316847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.316904 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.316908 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.316950 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.316993 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":23,"thread_start_us":118,"threads_started":1}
I20260620 21:59:11.317306 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.312756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.6.241.129:44265)
0620 21:59:11.313190 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.313194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.313204 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:11.313248 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.313250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.313265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.313372 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:11.313536 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.314371 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.315955 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.316759 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.316850 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.316936 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.316969 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.317009 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":83,"threads_started":1}
I20260620 21:59:11.317962 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.312951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.6.241.130:44769)
0620 21:59:11.313291 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.313297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.313310 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:59:11.313355 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.313358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.313375 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.313597 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:11.313663 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.314483 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.315937 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.316759 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.317727 (+ 968us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.317814 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.317841 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.317885 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":122,"threads_started":1}
I20260620 21:59:11.318102 14122 tablet_service.cc:1467] Processing CreateTablet for tablet a1f85c03ffcc4705ab163da984e12e83 (DEFAULT_TABLE table=test-table [id=5df82ed6daf34bc783a16c004fc50350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:11.318475 14122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f85c03ffcc4705ab163da984e12e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:11.318677 14259 tablet_service.cc:1467] Processing CreateTablet for tablet a1f85c03ffcc4705ab163da984e12e83 (DEFAULT_TABLE table=test-table [id=5df82ed6daf34bc783a16c004fc50350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:11.318982 14259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f85c03ffcc4705ab163da984e12e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:11.321628 14491 tablet_bootstrap.cc:492] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: Bootstrap starting.
I20260620 21:59:11.321918 14492 tablet_bootstrap.cc:492] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: Bootstrap starting.
I20260620 21:59:11.322422 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.312554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:41127 (local address 127.0.0.1:44776)
0620 21:59:11.314484 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.314494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.314564 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.314807 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.314809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.314811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:11.314875 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.314877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.315899 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.315900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.316645 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.316648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.322075 (+ 5427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.322076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.322166 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.322172 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.322259 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.322330 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1910,"thread_start_us":105,"threads_started":1}
I20260620 21:59:11.322454 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.312828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.6.241.131:41127)
0620 21:59:11.312989 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.312995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.314567 (+ 1572us) server_negotiation.cc:407] Connection header received
0620 21:59:11.314687 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.314690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.314712 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.314811 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:11.314945 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.315814 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.316735 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.317502 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.317571 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.317643 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.322302 (+ 4659us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.322373 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260620 21:59:11.322382 14491 tablet_bootstrap.cc:654] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:11.322906 14491 log.cc:824] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:11.322961 14492 tablet_bootstrap.cc:654] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:11.323297 14492 log.cc:824] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:11.323256 14395 tablet_service.cc:1467] Processing CreateTablet for tablet a1f85c03ffcc4705ab163da984e12e83 (DEFAULT_TABLE table=test-table [id=5df82ed6daf34bc783a16c004fc50350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:11.323592 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f85c03ffcc4705ab163da984e12e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:11.323781 14491 tablet_bootstrap.cc:492] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: No bootstrap required, opened a new log
I20260620 21:59:11.323848 14491 ts_tablet_manager.cc:1397] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:11.324082 14492 tablet_bootstrap.cc:492] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: No bootstrap required, opened a new log
I20260620 21:59:11.324164 14492 ts_tablet_manager.cc:1397] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:11.325625 14492 raft_consensus.cc:348] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } }
I20260620 21:59:11.325783 14492 raft_consensus.cc:374] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:11.325825 14492 raft_consensus.cc:729] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9da6d6ed39410d914401852c1b60a5, State: Initialized, Role: FOLLOWER
I20260620 21:59:11.325964 14492 consensus_queue.cc:260] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } }
I20260620 21:59:11.326269 14189 heartbeater.cc:503] Master 127.6.241.190:43375 was elected leader, sending a full tablet report...
I20260620 21:59:11.326473 14491 raft_consensus.cc:348] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } }
I20260620 21:59:11.326669 14491 raft_consensus.cc:374] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:11.326713 14491 raft_consensus.cc:729] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e1b09488874feab266f10e487666cc, State: Initialized, Role: FOLLOWER
I20260620 21:59:11.326867 14491 consensus_queue.cc:260] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } }
I20260620 21:59:11.327171 14326 heartbeater.cc:503] Master 127.6.241.190:43375 was elected leader, sending a full tablet report...
I20260620 21:59:11.327350 14491 ts_tablet_manager.cc:1428] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 21:59:11.328434 14492 ts_tablet_manager.cc:1428] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 21:59:11.328828 14497 tablet_bootstrap.cc:492] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: Bootstrap starting.
I20260620 21:59:11.329481 14497 tablet_bootstrap.cc:654] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:11.330191 14497 log.cc:824] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:11.331130 14497 tablet_bootstrap.cc:492] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: No bootstrap required, opened a new log
I20260620 21:59:11.331216 14497 ts_tablet_manager.cc:1397] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:11.332715 14497 raft_consensus.cc:348] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } }
I20260620 21:59:11.332878 14497 raft_consensus.cc:374] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:11.332901 14497 raft_consensus.cc:729] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8765db7344cf180a4df768e8623be, State: Initialized, Role: FOLLOWER
I20260620 21:59:11.333038 14497 consensus_queue.cc:260] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } }
I20260620 21:59:11.333316 14497 ts_tablet_manager.cc:1428] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:11.333392 14462 heartbeater.cc:503] Master 127.6.241.190:43375 was elected leader, sending a full tablet report...
I20260620 21:59:11.380077 14499 raft_consensus.cc:482] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:11.380263 14499 raft_consensus.cc:504] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } }
I20260620 21:59:11.380716 14499 leader_election.cc:288] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e1b09488874feab266f10e487666cc (127.6.241.130:44769), da9da6d6ed39410d914401852c1b60a5 (127.6.241.129:44265)
I20260620 21:59:11.384578 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.380935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:44265 (local address 127.6.241.131:51463)
0620 21:59:11.380999 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.381008 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.381065 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.381298 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.381298 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.381299 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:11.381341 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.381342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.382311 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.382313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.383722 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.383723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.384304 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.384306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.384389 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.384393 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.384450 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.384487 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":62}
I20260620 21:59:11.384593 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.381014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:51463 (local address 127.6.241.129:44265)
0620 21:59:11.381068 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.381070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.381082 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:59:11.381130 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.381131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.381145 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.381264 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:11.381410 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.382207 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.383860 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.384222 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.384281 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.384288 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.384471 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.384519 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 21:59:11.384712 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.380967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:33457 (local address 127.6.241.130:44769)
0620 21:59:11.381091 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.381094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.381102 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:11.381130 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.381131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.381142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.381235 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:11.381411 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.382191 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.383829 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.384309 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.384391 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.384399 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.384585 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.384638 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 21:59:11.384992 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.380907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:44769 (local address 127.6.241.131:33457)
0620 21:59:11.380973 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.380985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:11.381029 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:11.381239 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:11.381239 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:11.381241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:11.381316 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.381318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.382311 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.382313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.383714 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:11.383718 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.384394 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.384395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:11.384492 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.384496 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:11.384551 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 21:59:11.384591 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":100}
I20260620 21:59:11.385214 14142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f85c03ffcc4705ab163da984e12e83" candidate_uuid: "60e8765db7344cf180a4df768e8623be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9da6d6ed39410d914401852c1b60a5" is_pre_election: true
I20260620 21:59:11.385388 14142 raft_consensus.cc:2454] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60e8765db7344cf180a4df768e8623be in term 0.
I20260620 21:59:11.385373 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f85c03ffcc4705ab163da984e12e83" candidate_uuid: "60e8765db7344cf180a4df768e8623be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e1b09488874feab266f10e487666cc" is_pre_election: true
I20260620 21:59:11.385551 14279 raft_consensus.cc:2454] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60e8765db7344cf180a4df768e8623be in term 0.
I20260620 21:59:11.385663 14348 leader_election.cc:302] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60e8765db7344cf180a4df768e8623be, da9da6d6ed39410d914401852c1b60a5; no voters:
I20260620 21:59:11.385797 14499 raft_consensus.cc:2781] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:11.385855 14499 raft_consensus.cc:482] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:11.385871 14499 raft_consensus.cc:3037] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:11.386705 14499 raft_consensus.cc:504] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } }
I20260620 21:59:11.386904 14499 leader_election.cc:288] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [CANDIDATE]: Term 1 election: Requested vote from peers 93e1b09488874feab266f10e487666cc (127.6.241.130:44769), da9da6d6ed39410d914401852c1b60a5 (127.6.241.129:44265)
I20260620 21:59:11.386996 14142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f85c03ffcc4705ab163da984e12e83" candidate_uuid: "60e8765db7344cf180a4df768e8623be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9da6d6ed39410d914401852c1b60a5"
I20260620 21:59:11.387034 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f85c03ffcc4705ab163da984e12e83" candidate_uuid: "60e8765db7344cf180a4df768e8623be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e1b09488874feab266f10e487666cc"
I20260620 21:59:11.387066 14142 raft_consensus.cc:3037] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:11.387138 14279 raft_consensus.cc:3037] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:11.387899 14279 raft_consensus.cc:2454] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60e8765db7344cf180a4df768e8623be in term 1.
I20260620 21:59:11.387902 14142 raft_consensus.cc:2454] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60e8765db7344cf180a4df768e8623be in term 1.
I20260620 21:59:11.388120 14349 leader_election.cc:302] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60e8765db7344cf180a4df768e8623be, 93e1b09488874feab266f10e487666cc; no voters:
I20260620 21:59:11.388214 14499 raft_consensus.cc:2781] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:11.388381 14499 raft_consensus.cc:686] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 1 LEADER]: Becoming Leader. State: Replica: 60e8765db7344cf180a4df768e8623be, State: Running, Role: LEADER
I20260620 21:59:11.388504 14499 consensus_queue.cc:237] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } }
I20260620 21:59:11.389420 13224 catalog_manager.cc:5495] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be reported cstate change: term changed from 0 to 1, leader changed from <none> to 60e8765db7344cf180a4df768e8623be (127.6.241.131). New cstate: current_term: 1 leader_uuid: "60e8765db7344cf180a4df768e8623be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } health_report { overall_health: HEALTHY } } }
W20260620 21:59:11.392465 14328 tablet.cc:2369] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:11.412549 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.410156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.6.241.131:41127)
0620 21:59:11.410230 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.410231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.410240 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:11.410275 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.410276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.410294 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.410358 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:11.410491 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.411015 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.411828 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.412058 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.412111 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.412225 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:11.412331 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:11.412388 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.412419 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.412438 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:59:11.414543 14142 raft_consensus.cc:1261] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [term 1 FOLLOWER]: Refusing update from remote peer 60e8765db7344cf180a4df768e8623be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:11.414557 14279 raft_consensus.cc:1261] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [term 1 FOLLOWER]: Refusing update from remote peer 60e8765db7344cf180a4df768e8623be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:11.414971 14500 consensus_queue.cc:1035] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:11.415110 14499 consensus_queue.cc:1035] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [LEADER]: Connected to new peer: Peer: permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:11.419070 14505 mvcc.cc:205] Tried to move back new op lower bound from 7299042309791244288 to 7299042309688107008. Current Snapshot: MvccSnapshot[applied={T|T < 7299042309791244288 or (T in {7299042309791244288})}]
I20260620 21:59:11.420475 13224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:11.420565 13224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:11.421941 13224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 782f2eef2b6e43c3a1f1122b1598d935: Sending DeleteTablet for 3 replicas of tablet a1f85c03ffcc4705ab163da984e12e83
I20260620 21:59:11.422276 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f85c03ffcc4705ab163da984e12e83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:11 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:44776
I20260620 21:59:11.422293 14122 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f85c03ffcc4705ab163da984e12e83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:11 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:35518
I20260620 21:59:11.422524 14512 tablet_replica.cc:330] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: stopping tablet replica
I20260620 21:59:11.422664 14512 raft_consensus.cc:2229] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:11.422744 14259 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f85c03ffcc4705ab163da984e12e83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:11 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:59804
I20260620 21:59:11.422797 14512 raft_consensus.cc:2258] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:11.422950 14513 tablet_replica.cc:330] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: stopping tablet replica
I20260620 21:59:11.423032 14513 raft_consensus.cc:2229] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:11.423169 14513 raft_consensus.cc:2258] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:11.423662 14513 ts_tablet_manager.cc:1905] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:11.424217 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 13193
I20260620 21:59:11.425889 14513 ts_tablet_manager.cc:1918] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:11.425956 14513 log.cc:1196] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/a1f85c03ffcc4705ab163da984e12e83
I20260620 21:59:11.426214 14513 ts_tablet_manager.cc:1939] T a1f85c03ffcc4705ab163da984e12e83 P 60e8765db7344cf180a4df768e8623be: Deleting consensus metadata
I20260620 21:59:11.431208 14512 ts_tablet_manager.cc:1905] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:59:11.431272 14350 connection.cc:535] server connection from 127.0.0.1:44776 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:59:11.432627 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.423467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.6.241.188:46449)
0620 21:59:11.423509 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.423511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.423519 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:11.423539 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.423540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.423552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.423614 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.424359 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.425486 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.426618 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.426956 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.427026 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.429058 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.429060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.429060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.429069 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.431006 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.431656 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.431657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.431658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.431734 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.431954 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.431955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.431956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.432099 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.432182 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.432401 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.432504 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 21:59:11.433494 14512 ts_tablet_manager.cc:1918] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:11.433648 14512 log.cc:1196] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/a1f85c03ffcc4705ab163da984e12e83
I20260620 21:59:11.434042 14512 ts_tablet_manager.cc:1939] T a1f85c03ffcc4705ab163da984e12e83 P da9da6d6ed39410d914401852c1b60a5: Deleting consensus metadata
W20260620 21:59:11.434469 14077 connection.cc:441] server connection from 127.0.0.1:35518 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35518 (request call id 1) could send its response
I20260620 21:59:11.434736 14514 tablet_replica.cc:330] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: stopping tablet replica
I20260620 21:59:11.434914 14514 raft_consensus.cc:2229] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:11.435099 14514 raft_consensus.cc:2258] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:11.436163 14514 ts_tablet_manager.cc:1905] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:11.438390 14514 ts_tablet_manager.cc:1918] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:11.438524 14514 log.cc:1196] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/a1f85c03ffcc4705ab163da984e12e83
I20260620 21:59:11.438918 14514 ts_tablet_manager.cc:1939] T a1f85c03ffcc4705ab163da984e12e83 P 93e1b09488874feab266f10e487666cc: Deleting consensus metadata
W20260620 21:59:11.439483 14214 connection.cc:441] server connection from 127.0.0.1:59804 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59804 (request call id 1) could send its response
I20260620 21:59:11.442771 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.433721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.6.241.188:46449)
0620 21:59:11.433762 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.433764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.434044 (+ 280us) server_negotiation.cc:407] Connection header received
0620 21:59:11.434148 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.434149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.434159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.434189 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.434941 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.435722 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.437165 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.437447 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.437485 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.439667 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.439669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.439670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.439677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.441426 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.441930 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.441931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.441931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.441995 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.442181 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.442182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.442182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.442304 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.442413 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.442544 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.442668 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 21:59:11.443274 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.433999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.6.241.189:32981)
0620 21:59:11.434041 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:11.434044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:11.434196 (+ 152us) server_negotiation.cc:407] Connection header received
0620 21:59:11.434294 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:11.434295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:11.434310 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:11.434339 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:11.435022 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.435749 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.437073 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:11.437394 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:11.437436 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:11.439765 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:11.439767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:11.439767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:11.439774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:11.441604 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.442032 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.442033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.442034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.442104 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:11.442587 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:11.442588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:11.442589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:11.442771 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:11.442844 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:11.443050 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 21:59:11.443163 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:59:12.419308 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.418880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.130:37101)
0620 21:59:12.419143 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.419228 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260620 21:59:12.419335 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.418881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:41277)
0620 21:59:12.419057 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.419271 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
W20260620 21:59:12.419528 14462 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:43375 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
W20260620 21:59:12.419528 14326 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:43375 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
I20260620 21:59:12.419535 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.419220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.129:59191)
0620 21:59:12.419424 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.419495 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":54,"threads_started":1}
W20260620 21:59:12.419907 14189 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:43375 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
I20260620 21:59:12.922565 14525 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 782f2eef2b6e43c3a1f1122b1598d935)
I20260620 21:59:12.922678 14526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 782f2eef2b6e43c3a1f1122b1598d935)
I20260620 21:59:12.922755 14525 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } }
I20260620 21:59:12.922822 14526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } }
I20260620 21:59:12.923074 14526 leader_election.cc:288] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375), d4d0aab8261446af860d68672e92d0de (127.6.241.189:32981)
I20260620 21:59:12.923064 14525 leader_election.cc:288] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375), 9543330b81d8478990d6e37386518b81 (127.6.241.188:46449)
I20260620 21:59:12.923390 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.923167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60472)
0620 21:59:12.923291 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.923343 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260620 21:59:12.923524 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.923163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60468)
0620 21:59:12.923451 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.923490 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":47,"threads_started":1}
W20260620 21:59:12.923597 13996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
W20260620 21:59:12.923663 13810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
I20260620 21:59:12.924027 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.923944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60484)
0620 21:59:12.923978 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.924001 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 21:59:12.924044 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.923966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60488)
0620 21:59:12.924002 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.924021 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 21:59:12.924111 13996 leader_election.cc:334] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375): Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
W20260620 21:59:12.924213 13810 leader_election.cc:334] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375): Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
I20260620 21:59:12.927172 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.923382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.6.241.189:32981)
0620 21:59:12.923630 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:12.923632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:12.923640 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:12.923662 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:12.923663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:12.923674 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:12.923739 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:12.923953 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:12.924780 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:12.926230 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:12.926818 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:12.926895 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:12.926906 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:12.927041 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 21:59:12.927086 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":232,"thread_start_us":39,"threads_started":1}
I20260620 21:59:12.927167 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.923270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:32981 (local address 127.0.0.1:52096)
0620 21:59:12.923446 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.923456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:12.923511 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:12.923790 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:12.923791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:12.923793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:12.923893 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:12.923895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:12.924880 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:12.924882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:12.926135 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:12.926138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:12.926896 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:12.926897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:12.926963 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:12.926967 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:12.927027 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 21:59:12.927073 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":44,"thread_start_us":34,"threads_started":1}
I20260620 21:59:12.927172 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.923460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.6.241.188:46449)
0620 21:59:12.923547 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:12.923548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:12.923703 (+ 155us) server_negotiation.cc:407] Connection header received
0620 21:59:12.923724 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:12.923725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:12.923734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:12.923786 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:12.924086 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:12.924801 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:12.926230 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:12.926818 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:12.926895 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:12.926906 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:12.927041 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 21:59:12.927086 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":72,"thread_start_us":50,"threads_started":1}
I20260620 21:59:12.927278 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.923347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:46449 (local address 127.0.0.1:46758)
0620 21:59:12.923587 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.923597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:12.923650 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:12.923977 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:12.923978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:12.923980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:12.924016 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:12.924017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:12.924881 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:12.924881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:12.926136 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:12.926138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:12.926896 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:12.926897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:12.926963 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:12.926967 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:12.927027 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 21:59:12.927079 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":6,"thread_start_us":53,"threads_started":1}
I20260620 21:59:12.927618 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9543330b81d8478990d6e37386518b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d4d0aab8261446af860d68672e92d0de" is_pre_election: true
I20260620 21:59:12.927654 14019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4d0aab8261446af860d68672e92d0de" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9543330b81d8478990d6e37386518b81" is_pre_election: true
I20260620 21:59:12.927729 13833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9543330b81d8478990d6e37386518b81 in term 1.
I20260620 21:59:12.927778 14019 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4d0aab8261446af860d68672e92d0de in term 1.
I20260620 21:59:12.927917 13810 leader_election.cc:302] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9543330b81d8478990d6e37386518b81, d4d0aab8261446af860d68672e92d0de; no voters: 782f2eef2b6e43c3a1f1122b1598d935
I20260620 21:59:12.927937 13995 leader_election.cc:302] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9543330b81d8478990d6e37386518b81, d4d0aab8261446af860d68672e92d0de; no voters: 782f2eef2b6e43c3a1f1122b1598d935
I20260620 21:59:12.928035 14526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 21:59:12.928035 14525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 21:59:12.928071 14526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 782f2eef2b6e43c3a1f1122b1598d935)
I20260620 21:59:12.928071 14525 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Starting leader election (detected failure of leader 782f2eef2b6e43c3a1f1122b1598d935)
I20260620 21:59:12.928128 14525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:59:12.928128 14526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:59:12.929000 14526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } }
I20260620 21:59:12.929000 14525 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } }
I20260620 21:59:12.929147 14526 leader_election.cc:288] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 2 election: Requested vote from peers 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375), d4d0aab8261446af860d68672e92d0de (127.6.241.189:32981)
I20260620 21:59:12.929275 14525 leader_election.cc:288] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [CANDIDATE]: Term 2 election: Requested vote from peers 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375), 9543330b81d8478990d6e37386518b81 (127.6.241.188:46449)
I20260620 21:59:12.929315 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9543330b81d8478990d6e37386518b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d4d0aab8261446af860d68672e92d0de"
I20260620 21:59:12.929409 13833 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9543330b81d8478990d6e37386518b81 in current term 2: Already voted for candidate d4d0aab8261446af860d68672e92d0de in this term.
I20260620 21:59:12.929410 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.929222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60490)
0620 21:59:12.929301 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.929335 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 21:59:12.929455 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.929271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60500)
0620 21:59:12.929394 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.929421 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 21:59:12.929631 14019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4d0aab8261446af860d68672e92d0de" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9543330b81d8478990d6e37386518b81"
I20260620 21:59:12.929699 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.929610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60504)
0620 21:59:12.929646 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.929675 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 21:59:12.929782 14019 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4d0aab8261446af860d68672e92d0de in current term 2: Already voted for candidate 9543330b81d8478990d6e37386518b81 in this term.
W20260620 21:59:12.929782 13810 leader_election.cc:334] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375): Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
I20260620 21:59:12.929705 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.929640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60508)
0620 21:59:12.929672 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.929689 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 21:59:12.929894 13996 leader_election.cc:334] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375): Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
I20260620 21:59:12.929937 13996 leader_election.cc:302] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9543330b81d8478990d6e37386518b81; no voters: 782f2eef2b6e43c3a1f1122b1598d935, d4d0aab8261446af860d68672e92d0de
I20260620 21:59:12.929956 13810 leader_election.cc:302] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4d0aab8261446af860d68672e92d0de; no voters: 782f2eef2b6e43c3a1f1122b1598d935, 9543330b81d8478990d6e37386518b81
I20260620 21:59:12.929991 14526 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 21:59:12.930022 14525 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 21:59:13.420307 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:13.420022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:48273)
0620 21:59:13.420201 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:13.420253 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260620 21:59:13.420305 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:13.420022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.130:36907)
0620 21:59:13.420199 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:13.420244 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260620 21:59:13.420629 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:13.420315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.129:43043)
0620 21:59:13.420551 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:13.420585 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":47,"threads_started":1}
I20260620 21:59:14.421438 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.421089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.130:43915)
0620 21:59:14.421322 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.421375 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260620 21:59:14.421511 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.421089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.129:36869)
0620 21:59:14.421419 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.421462 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":75,"threads_started":1}
I20260620 21:59:14.421959 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.421770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:42573)
0620 21:59:14.421872 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.421914 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
I20260620 21:59:14.825289 14541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:14.825402 14541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } }
I20260620 21:59:14.825605 14541 leader_election.cc:288] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375), d4d0aab8261446af860d68672e92d0de (127.6.241.189:32981)
I20260620 21:59:14.825891 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9543330b81d8478990d6e37386518b81" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d4d0aab8261446af860d68672e92d0de" is_pre_election: true
I20260620 21:59:14.825995 13833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9543330b81d8478990d6e37386518b81 in term 2.
I20260620 21:59:14.825977 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.825762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60772)
0620 21:59:14.825902 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.825938 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260620 21:59:14.826159 13995 leader_election.cc:302] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9543330b81d8478990d6e37386518b81, d4d0aab8261446af860d68672e92d0de; no voters:
I20260620 21:59:14.826275 14541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 21:59:14.826325 14541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:14.826373 14541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 2 FOLLOWER]: Advancing to term 3
I20260620 21:59:14.826542 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.826448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60774)
0620 21:59:14.826490 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.826523 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 21:59:14.826642 13996 leader_election.cc:334] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375): Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
I20260620 21:59:14.827219 14541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } }
I20260620 21:59:14.827368 14541 leader_election.cc:288] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 3 election: Requested vote from peers 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375), d4d0aab8261446af860d68672e92d0de (127.6.241.189:32981)
I20260620 21:59:14.827526 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9543330b81d8478990d6e37386518b81" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d4d0aab8261446af860d68672e92d0de"
I20260620 21:59:14.827566 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.827477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60790)
0620 21:59:14.827513 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.827539 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 21:59:14.827618 13833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 2 FOLLOWER]: Advancing to term 3
I20260620 21:59:14.827858 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.827772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60804)
0620 21:59:14.827816 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.827839 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 21:59:14.827932 13996 leader_election.cc:334] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375): Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
I20260620 21:59:14.828389 13833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9543330b81d8478990d6e37386518b81 in term 3.
I20260620 21:59:14.828567 13995 leader_election.cc:302] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9543330b81d8478990d6e37386518b81, d4d0aab8261446af860d68672e92d0de; no voters: 782f2eef2b6e43c3a1f1122b1598d935
I20260620 21:59:14.828665 14541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 21:59:14.828776 14541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [term 3 LEADER]: Becoming Leader. State: Replica: 9543330b81d8478990d6e37386518b81, State: Running, Role: LEADER
I20260620 21:59:14.828873 14541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } }
I20260620 21:59:14.829222 14544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9543330b81d8478990d6e37386518b81. Latest consensus state: current_term: 3 leader_uuid: "9543330b81d8478990d6e37386518b81" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:14.829301 14544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:14.829457 14546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:14.829937 14546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:14.830034 14546 catalog_manager.cc:1259] Loaded cluster ID: 64f670dccdbc4b8cbbf599ac69360f5d
I20260620 21:59:14.830065 14546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:14.830252 14546 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:14.830349 14546 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Loaded TSK: 0
I20260620 21:59:14.830500 14546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:14.850922 14009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:14.852583 14009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:14.853641 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.853529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60824)
0620 21:59:14.853581 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.853613 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 21:59:14.853693 13833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [term 3 FOLLOWER]: Refusing update from remote peer 9543330b81d8478990d6e37386518b81: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 21:59:14.853917 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.853846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.0.0.1:60836)
0620 21:59:14.853873 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.853895 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 21:59:14.853983 14544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 21:59:14.854128 13996 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 -> Peer 782f2eef2b6e43c3a1f1122b1598d935 (127.6.241.190:43375): Couldn't send request to peer 782f2eef2b6e43c3a1f1122b1598d935. Status: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 21:59:14.854578 14548 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9543330b81d8478990d6e37386518b81. Latest consensus state: current_term: 3 leader_uuid: "9543330b81d8478990d6e37386518b81" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:14.854713 14548 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:14.855098 14544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9543330b81d8478990d6e37386518b81" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:14.855182 14544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:14.855472 14549 mvcc.cc:205] Tried to move back new op lower bound from 7299042323878506496 to 7299042323779637248. Current Snapshot: MvccSnapshot[applied={T|T < 7299042323878506496}]
I20260620 21:59:14.855777 14548 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9543330b81d8478990d6e37386518b81" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "782f2eef2b6e43c3a1f1122b1598d935" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43375 } } peers { permanent_uuid: "d4d0aab8261446af860d68672e92d0de" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 32981 } attrs { promote: false } } peers { permanent_uuid: "9543330b81d8478990d6e37386518b81" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 46449 } attrs { promote: false } } }
I20260620 21:59:14.855974 14548 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0aab8261446af860d68672e92d0de [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:14.860348 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.856922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:44769 (local address 127.0.0.1:59808)
0620 21:59:14.857074 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.857083 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:14.857125 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:14.857524 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:14.857525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:14.857527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:14.857565 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:14.857567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.858445 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.858446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:14.859499 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:14.859502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.860047 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.860048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:14.860113 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:14.860117 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:14.860166 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 21:59:14.860210 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":114,"threads_started":1}
I20260620 21:59:14.860400 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.856807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:41127 (local address 127.0.0.1:44790)
0620 21:59:14.856839 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.856851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:14.856901 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:14.857125 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:14.857126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:14.857128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:14.857191 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:14.857193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.858153 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.858154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:14.859401 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:14.859403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.860123 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.860124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:14.860184 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:14.860186 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:59:14.860236 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 21:59:14.860269 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 21:59:14.860399 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.856879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.6.241.131:41127)
0620 21:59:14.856994 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:14.856996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:14.857006 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:14.857032 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:14.857033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:14.857045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:14.857125 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:14.857294 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.858080 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.859492 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.860058 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.860126 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:14.860137 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:14.860248 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 21:59:14.860295 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260620 21:59:14.860613 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.856926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:44265 (local address 127.0.0.1:35534)
0620 21:59:14.857194 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.857203 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:14.857238 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:14.857611 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:14.857612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:14.857614 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:14.857659 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:14.857660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.858549 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.858550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:14.859650 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:14.859652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.860407 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.860408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:14.860473 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:14.860477 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:14.860509 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 21:59:14.860547 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260620 21:59:14.860857 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.857276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.6.241.130:44769)
0620 21:59:14.857385 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:14.857388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:14.857395 (+ 7us) server_negotiation.cc:407] Connection header received
0620 21:59:14.857418 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:14.857418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:14.857428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:14.857620 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:14.857655 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.858374 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.859570 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.859958 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.860038 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:14.860049 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:14.860737 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 21:59:14.860790 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":60,"threads_started":1}
I20260620 21:59:14.860900 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.857168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.6.241.129:44265)
0620 21:59:14.857458 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:14.857460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:14.857468 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:14.857494 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:14.857496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:14.857506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:14.857604 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:14.857714 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.858450 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.859739 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:14.860317 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:14.860734 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:14.860750 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:14.860774 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 21:59:14.860822 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":60,"threads_started":1}
I20260620 21:59:14.860960 14395 tablet_service.cc:1467] Processing CreateTablet for tablet 354e01fff5324ecd92cb6ad7ef97d654 (DEFAULT_TABLE table=test-table [id=197d031f576443459ca536be72f272ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:14.861086 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354e01fff5324ecd92cb6ad7ef97d654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:14.861184 14259 tablet_service.cc:1467] Processing CreateTablet for tablet 354e01fff5324ecd92cb6ad7ef97d654 (DEFAULT_TABLE table=test-table [id=197d031f576443459ca536be72f272ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:14.861301 14259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354e01fff5324ecd92cb6ad7ef97d654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:14.861275 14122 tablet_service.cc:1467] Processing CreateTablet for tablet 354e01fff5324ecd92cb6ad7ef97d654 (DEFAULT_TABLE table=test-table [id=197d031f576443459ca536be72f272ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:14.861397 14122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354e01fff5324ecd92cb6ad7ef97d654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:14.862468 14558 tablet_bootstrap.cc:492] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5: Bootstrap starting.
I20260620 21:59:14.862651 14559 tablet_bootstrap.cc:492] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: Bootstrap starting.
I20260620 21:59:14.862772 14560 tablet_bootstrap.cc:492] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: Bootstrap starting.
I20260620 21:59:14.863058 14558 tablet_bootstrap.cc:654] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:14.863185 14560 tablet_bootstrap.cc:654] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:14.863672 14558 tablet_bootstrap.cc:492] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5: No bootstrap required, opened a new log
I20260620 21:59:14.863737 14559 tablet_bootstrap.cc:654] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:14.863725 14558 ts_tablet_manager.cc:1397] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:14.863744 14560 tablet_bootstrap.cc:492] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: No bootstrap required, opened a new log
I20260620 21:59:14.863782 14560 ts_tablet_manager.cc:1397] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:14.863889 14558 raft_consensus.cc:348] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } }
I20260620 21:59:14.863888 14560 raft_consensus.cc:348] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } }
I20260620 21:59:14.863955 14558 raft_consensus.cc:374] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:14.863955 14560 raft_consensus.cc:374] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:14.863976 14558 raft_consensus.cc:729] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9da6d6ed39410d914401852c1b60a5, State: Initialized, Role: FOLLOWER
I20260620 21:59:14.863976 14560 raft_consensus.cc:729] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8765db7344cf180a4df768e8623be, State: Initialized, Role: FOLLOWER
I20260620 21:59:14.864029 14558 consensus_queue.cc:260] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } }
I20260620 21:59:14.864040 14560 consensus_queue.cc:260] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } }
I20260620 21:59:14.864212 14560 ts_tablet_manager.cc:1428] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:14.864223 14558 ts_tablet_manager.cc:1428] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:14.864256 14559 tablet_bootstrap.cc:492] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: No bootstrap required, opened a new log
I20260620 21:59:14.864310 14559 ts_tablet_manager.cc:1397] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:14.864758 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.864596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.129:57543)
0620 21:59:14.864688 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.864726 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 21:59:14.864812 14188 heartbeater.cc:503] Master 127.6.241.188:46449 was elected leader, sending a full tablet report...
I20260620 21:59:14.865129 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.864997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.129:37121)
0620 21:59:14.865080 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.865105 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 21:59:14.865185 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.865008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:38155)
0620 21:59:14.865102 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.865135 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 21:59:14.864440 14559 raft_consensus.cc:348] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } }
I20260620 21:59:14.865432 14461 heartbeater.cc:503] Master 127.6.241.188:46449 was elected leader, sending a full tablet report...
I20260620 21:59:14.865533 14559 raft_consensus.cc:374] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:14.865592 14559 raft_consensus.cc:729] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e1b09488874feab266f10e487666cc, State: Initialized, Role: FOLLOWER
I20260620 21:59:14.865672 14559 consensus_queue.cc:260] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } }
I20260620 21:59:14.865973 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.865880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:59985)
0620 21:59:14.865915 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.865943 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 21:59:14.866160 14559 ts_tablet_manager.cc:1428] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:14.866376 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.866285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.130:58801)
0620 21:59:14.866319 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.866344 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 21:59:14.866745 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.866656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.130:43711)
0620 21:59:14.866688 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.866718 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 21:59:14.866753 14325 heartbeater.cc:503] Master 127.6.241.188:46449 was elected leader, sending a full tablet report...
W20260620 21:59:14.995114 14191 tablet.cc:2369] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:59:15.032886 14464 tablet.cc:2369] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:15.123871 14564 raft_consensus.cc:482] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:15.123980 14564 raft_consensus.cc:504] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } }
I20260620 21:59:15.124174 14564 leader_election.cc:288] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da9da6d6ed39410d914401852c1b60a5 (127.6.241.129:44265), 93e1b09488874feab266f10e487666cc (127.6.241.130:44769)
I20260620 21:59:15.124442 14142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354e01fff5324ecd92cb6ad7ef97d654" candidate_uuid: "60e8765db7344cf180a4df768e8623be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9da6d6ed39410d914401852c1b60a5" is_pre_election: true
I20260620 21:59:15.124467 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354e01fff5324ecd92cb6ad7ef97d654" candidate_uuid: "60e8765db7344cf180a4df768e8623be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e1b09488874feab266f10e487666cc" is_pre_election: true
I20260620 21:59:15.124550 14142 raft_consensus.cc:2454] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60e8765db7344cf180a4df768e8623be in term 0.
I20260620 21:59:15.124565 14279 raft_consensus.cc:2454] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60e8765db7344cf180a4df768e8623be in term 0.
I20260620 21:59:15.124742 14348 leader_election.cc:302] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60e8765db7344cf180a4df768e8623be, da9da6d6ed39410d914401852c1b60a5; no voters:
I20260620 21:59:15.124845 14564 raft_consensus.cc:2781] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:15.124902 14564 raft_consensus.cc:482] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:15.124917 14564 raft_consensus.cc:3037] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:15.125713 14564 raft_consensus.cc:504] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } }
I20260620 21:59:15.125866 14564 leader_election.cc:288] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [CANDIDATE]: Term 1 election: Requested vote from peers da9da6d6ed39410d914401852c1b60a5 (127.6.241.129:44265), 93e1b09488874feab266f10e487666cc (127.6.241.130:44769)
I20260620 21:59:15.126020 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354e01fff5324ecd92cb6ad7ef97d654" candidate_uuid: "60e8765db7344cf180a4df768e8623be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e1b09488874feab266f10e487666cc"
I20260620 21:59:15.126052 14142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354e01fff5324ecd92cb6ad7ef97d654" candidate_uuid: "60e8765db7344cf180a4df768e8623be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9da6d6ed39410d914401852c1b60a5"
I20260620 21:59:15.126109 14279 raft_consensus.cc:3037] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:15.126140 14142 raft_consensus.cc:3037] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:15.126771 14279 raft_consensus.cc:2454] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60e8765db7344cf180a4df768e8623be in term 1.
I20260620 21:59:15.126955 14142 raft_consensus.cc:2454] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60e8765db7344cf180a4df768e8623be in term 1.
I20260620 21:59:15.126974 14349 leader_election.cc:302] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60e8765db7344cf180a4df768e8623be, 93e1b09488874feab266f10e487666cc; no voters:
I20260620 21:59:15.127164 14564 raft_consensus.cc:2781] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:15.127317 14564 raft_consensus.cc:686] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 1 LEADER]: Becoming Leader. State: Replica: 60e8765db7344cf180a4df768e8623be, State: Running, Role: LEADER
I20260620 21:59:15.127363 14564 consensus_queue.cc:237] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } }
I20260620 21:59:15.127688 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.127581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:41089)
0620 21:59:15.127621 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.127655 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 21:59:15.128274 14009 catalog_manager.cc:5495] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be reported cstate change: term changed from 0 to 1, leader changed from <none> to 60e8765db7344cf180a4df768e8623be (127.6.241.131). New cstate: current_term: 1 leader_uuid: "60e8765db7344cf180a4df768e8623be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e8765db7344cf180a4df768e8623be" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 41127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 } health_report { overall_health: UNKNOWN } } }
I20260620 21:59:15.144317 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.142245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.6.241.131:41127)
0620 21:59:15.142322 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:15.142324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:15.142338 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:59:15.142362 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:15.142362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:15.142371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:15.142417 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:15.142555 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:15.143088 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:15.143806 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:15.143984 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:15.144033 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:15.144137 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:15.144161 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:15.144197 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:15.144241 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 21:59:15.144258 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:59:15.145310 14279 raft_consensus.cc:1261] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [term 1 FOLLOWER]: Refusing update from remote peer 60e8765db7344cf180a4df768e8623be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:15.145358 14142 raft_consensus.cc:1261] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [term 1 FOLLOWER]: Refusing update from remote peer 60e8765db7344cf180a4df768e8623be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:15.145545 14567 consensus_queue.cc:1035] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e1b09488874feab266f10e487666cc" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:15.145656 14564 consensus_queue.cc:1035] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [LEADER]: Connected to new peer: Peer: permanent_uuid: "da9da6d6ed39410d914401852c1b60a5" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:15.146361 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.146230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.130:41185)
0620 21:59:15.146284 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.146326 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 21:59:15.146922 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.146752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.129:39805)
0620 21:59:15.146859 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.146889 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 21:59:15.147454 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.147356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:40367)
0620 21:59:15.147397 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.147422 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 21:59:15.147655 14570 mvcc.cc:205] Tried to move back new op lower bound from 7299042325073121280 to 7299042325002678272. Current Snapshot: MvccSnapshot[applied={T|T < 7299042325073121280}]
I20260620 21:59:15.147864 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.147773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.129:35061)
0620 21:59:15.147810 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.147834 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 21:59:15.148133 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.148043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.131:45617)
0620 21:59:15.148079 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.148106 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 21:59:15.148463 14571 mvcc.cc:205] Tried to move back new op lower bound from 7299042325073121280 to 7299042325002678272. Current Snapshot: MvccSnapshot[applied={T|T < 7299042325073121280}]
I20260620 21:59:15.148921 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.148798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43375 (local address 127.6.241.130:59409)
0620 21:59:15.148856 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.148889 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 21:59:15.149650 14008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:15.149739 14008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:15.150964 14008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9543330b81d8478990d6e37386518b81: Sending DeleteTablet for 3 replicas of tablet 354e01fff5324ecd92cb6ad7ef97d654
I20260620 21:59:15.151227 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet 354e01fff5324ecd92cb6ad7ef97d654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:15 UTC) from {username='oryx', principal='oryx/127.6.241.188@KRBTEST.COM'} at 127.0.0.1:44790
I20260620 21:59:15.151263 14122 tablet_service.cc:1514] Processing DeleteTablet for tablet 354e01fff5324ecd92cb6ad7ef97d654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:15 UTC) from {username='oryx', principal='oryx/127.6.241.188@KRBTEST.COM'} at 127.0.0.1:35534
I20260620 21:59:15.151475 14579 tablet_replica.cc:330] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5: stopping tablet replica
I20260620 21:59:15.151489 14259 tablet_service.cc:1514] Processing DeleteTablet for tablet 354e01fff5324ecd92cb6ad7ef97d654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:15 UTC) from {username='oryx', principal='oryx/127.6.241.188@KRBTEST.COM'} at 127.0.0.1:59808
I20260620 21:59:15.151544 14579 raft_consensus.cc:2229] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:15.151624 14579 raft_consensus.cc:2258] T 354e01fff5324ecd92cb6ad7ef97d654 P da9da6d6ed39410d914401852c1b60a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:15.151639 14578 tablet_replica.cc:330] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: stopping tablet replica
I20260620 21:59:15.151664 14580 tablet_replica.cc:330] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: stopping tablet replica
I20260620 21:59:15.151736 14580 raft_consensus.cc:2229] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:15.151765 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 13970
I20260620 21:59:15.151909 14580 raft_consensus.cc:2258] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:15.152552 14580 ts_tablet_manager.cc:1905] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:15.152643 14578 raft_consensus.cc:2229] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:15.152710 14578 raft_consensus.cc:2258] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:15.152976 14578 ts_tablet_manager.cc:1905] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:15.154420 14578 ts_tablet_manager.cc:1918] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:15.154466 14578 log.cc:1196] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/354e01fff5324ecd92cb6ad7ef97d654
I20260620 21:59:15.154481 14580 ts_tablet_manager.cc:1918] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:15.154536 14580 log.cc:1196] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/354e01fff5324ecd92cb6ad7ef97d654
I20260620 21:59:15.154778 14578 ts_tablet_manager.cc:1939] T 354e01fff5324ecd92cb6ad7ef97d654 P 60e8765db7344cf180a4df768e8623be: Deleting consensus metadata
I20260620 21:59:15.154783 14580 ts_tablet_manager.cc:1939] T 354e01fff5324ecd92cb6ad7ef97d654 P 93e1b09488874feab266f10e487666cc: Deleting consensus metadata
I20260620 21:59:15.155169 13995 catalog_manager.cc:4841] TS 60e8765db7344cf180a4df768e8623be (127.6.241.131:41127): tablet 354e01fff5324ecd92cb6ad7ef97d654 (table test-table [id=197d031f576443459ca536be72f272ef]) successfully deleted
I20260620 21:59:15.155202 13994 catalog_manager.cc:4841] TS 93e1b09488874feab266f10e487666cc (127.6.241.130:44769): tablet 354e01fff5324ecd92cb6ad7ef97d654 (table test-table [id=197d031f576443459ca536be72f272ef]) successfully deleted
I20260620 21:59:15.158584 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.158463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:44265 (local address 127.0.0.1:35544)
0620 21:59:15.158516 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.158545 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 21:59:15.158635 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 14196
W20260620 21:59:15.158787 13993 catalog_manager.cc:4584] TS da9da6d6ed39410d914401852c1b60a5 (127.6.241.129:44265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 354e01fff5324ecd92cb6ad7ef97d654: Network error: Client connection negotiation failed: client connection to 127.6.241.129:44265: connect: Connection refused (error 111)
I20260620 21:59:15.163908 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 14332
I20260620 21:59:15.169629 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 13689
I20260620 21:59:15.174346 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 13877
2026-06-20T21:59:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6640 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:15 dist-test-slave-gl2g krb5kdc[14583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:15 dist-test-slave-gl2g krb5kdc[14583](info): set up 2 sockets
Jun 20 21:59:15 dist-test-slave-gl2g krb5kdc[14583](info): commencing operation
krb5kdc: starting...
W20260620 21:59:16.215077 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:59:16 dist-test-slave-gl2g krb5kdc[14583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.190.keytab.
Entry for principal oryx/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.190.keytab.
I20260620 21:59:16.333570 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33021
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.190.keytab
--principal=oryx/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:33021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:16.414804 14596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:16.415019 14596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:16.415040 14596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:16.416177 14596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:16.416215 14596 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:59:16.416229 14596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:16.416242 14596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:16.416254 14596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:16.416266 14596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:16.417711 14596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:33021
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.190.keytab
--principal=oryx/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:16.417985 14596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:16.418234 14596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:16.423111 14596 server_base.cc:1034] running on GCE node
W20260620 21:59:16.423062 14602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:16.423070 14601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:16.423070 14604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:16.423619 14596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:16.423919 14596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:16.425076 14596 hybrid_clock.cc:648] HybridClock initialized: now 1781992756425072 us; error 35 us; skew 500 ppm
Jun 20 21:59:16 dist-test-slave-gl2g krb5kdc[14583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:16.427618 14596 init.cc:377] Logged in from keytab as oryx/127.6.241.190@KRBTEST.COM (short username oryx)
I20260620 21:59:16.428088 14596 webserver.cc:460] Webserver started at http://127.6.241.190:39763/ using document root <none> and password file <none>
I20260620 21:59:16.428326 14596 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:16.428391 14596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:16.428536 14596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:16.429626 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "48803be0f4160c3cf288ce45c997b912"
server_key_iv: "ca8e1161ee194759f50f8520e39f959f"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.429988 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "48803be0f4160c3cf288ce45c997b912"
server_key_iv: "ca8e1161ee194759f50f8520e39f959f"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.431449 14596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 21:59:16.432329 14611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:16.432528 14596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:16.432621 14596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "48803be0f4160c3cf288ce45c997b912"
server_key_iv: "ca8e1161ee194759f50f8520e39f959f"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.432700 14596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:16.451759 14596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:16.454300 14596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:16.454479 14596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:16.459257 14596 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:33021
I20260620 21:59:16.459296 14673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:33021 every 8 connection(s)
I20260620 21:59:16.459671 14596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:16.460321 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:16.462757 14674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: Bootstrap starting.
I20260620 21:59:16.463392 14674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:16.463693 14674 log.cc:824] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:16.464443 14674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: No bootstrap required, opened a new log
I20260620 21:59:16.466524 14674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33021 } }
I20260620 21:59:16.466709 14674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:16.466734 14674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23722ee6c5124c6e9a6e8e3f4cc6a077, State: Initialized, Role: FOLLOWER
I20260620 21:59:16.466907 14674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33021 } }
I20260620 21:59:16.466982 14674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:16.467023 14674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:16.467072 14674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:16.467687 14674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33021 } }
I20260620 21:59:16.467782 14674 leader_election.cc:302] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23722ee6c5124c6e9a6e8e3f4cc6a077; no voters:
I20260620 21:59:16.468006 14674 leader_election.cc:288] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:16.468065 14677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:16.468318 14677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [term 1 LEADER]: Becoming Leader. State: Replica: 23722ee6c5124c6e9a6e8e3f4cc6a077, State: Running, Role: LEADER
I20260620 21:59:16.468385 14674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:16.468669 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 14596
I20260620 21:59:16.468770 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:16.468977 7110 external_mini_cluster.cc:1428] Setting key 62aa11cade3c2616d8a2e46fe3bd9338
I20260620 21:59:16.468420 14677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33021 } }
I20260620 21:59:16.469815 14679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23722ee6c5124c6e9a6e8e3f4cc6a077. Latest consensus state: current_term: 1 leader_uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33021 } } }
I20260620 21:59:16.470104 14679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:16.469966 14678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23722ee6c5124c6e9a6e8e3f4cc6a077" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33021 } } }
W20260620 21:59:16.470288 14693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 21:59:16.470351 14678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077 [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:16.470360 14693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:16.470477 14695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:16.470677 14695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:16.472396 14695 catalog_manager.cc:1347] Generated new cluster ID: 30614cad9d3549d4a881400a3a6c7e68
I20260620 21:59:16.472450 14695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 21:59:16 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:16.481287 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.469628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.6.241.190:33021)
0620 21:59:16.469790 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:16.469803 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:16.470074 (+ 271us) server_negotiation.cc:407] Connection header received
0620 21:59:16.470220 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:16.470229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:16.470292 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:16.470447 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:16.471386 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.472630 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.473666 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.474087 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.474179 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:16.477033 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:16.477038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:16.477038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:16.477051 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:16.479571 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.480012 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.480015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.480016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.480130 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.480338 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.480339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.480339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.480637 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:16.480785 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:16.480908 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 21:59:16.481046 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260620 21:59:16.484221 14695 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:16.484818 14695 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:16.496394 14695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: Generated new TSK 0
I20260620 21:59:16.496630 14695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab.
Entry for principal oryx/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab.
I20260620 21:59:16.529197 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33021
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab
--principal=oryx/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:16.614038 14700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:16.614277 14700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:16.614306 14700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:16.615520 14700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:16.615563 14700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:16.615584 14700 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:59:16.615602 14700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:16.617108 14700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab
--principal=oryx/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.241.190:33021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:16.617333 14700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:16.617530 14700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:16.622869 14706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:16.622871 14708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:16.622871 14705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:16.623132 14700 server_base.cc:1034] running on GCE node
I20260620 21:59:16.623414 14700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:16.623677 14700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:16.624845 14700 hybrid_clock.cc:648] HybridClock initialized: now 1781992756624825 us; error 35 us; skew 500 ppm
Jun 20 21:59:16 dist-test-slave-gl2g krb5kdc[14583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:16.627533 14700 init.cc:377] Logged in from keytab as oryx/127.6.241.129@KRBTEST.COM (short username oryx)
I20260620 21:59:16.627954 14700 webserver.cc:460] Webserver started at http://127.6.241.129:41369/ using document root <none> and password file <none>
I20260620 21:59:16.628155 14700 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:16.628196 14700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:16.628295 14700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:16.629390 14700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "f36f11e47daa42778564432d5bd92867"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "add1120c38226fbbaa91b0aa19c0f7ce"
server_key_iv: "2cdc9b93fc2af6c9a65e98520455734c"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.629732 14700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "f36f11e47daa42778564432d5bd92867"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "add1120c38226fbbaa91b0aa19c0f7ce"
server_key_iv: "2cdc9b93fc2af6c9a65e98520455734c"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.631148 14700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 21:59:16.631899 14715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:16.632061 14700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:16.632160 14700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "f36f11e47daa42778564432d5bd92867"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "add1120c38226fbbaa91b0aa19c0f7ce"
server_key_iv: "2cdc9b93fc2af6c9a65e98520455734c"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.632232 14700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:16.652889 14700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:16.655790 14700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:16.655930 14700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:16.656461 14700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:16.656507 14700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:16.656533 14700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:16.656545 14700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:16.663863 14700 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:43489
I20260620 21:59:16.663946 14830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:43489 every 8 connection(s)
I20260620 21:59:16.664328 14700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:59:16 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:16.664568 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 14700
I20260620 21:59:16.664673 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:16.664918 7110 external_mini_cluster.cc:1428] Setting key 87fb38261208459180bb9a8033eadde4
I20260620 21:59:16.668972 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.656983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:55173 (local address 127.6.241.190:33021)
0620 21:59:16.657057 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:16.657061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:16.657253 (+ 192us) server_negotiation.cc:407] Connection header received
0620 21:59:16.657751 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:16.657753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:16.657766 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:16.657847 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:16.658903 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.659954 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.661762 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.662100 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.662193 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:16.665036 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:16.665038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:16.665038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:16.665048 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:16.667105 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.667722 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.667724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.667724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.667800 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.668116 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.668117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.668118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.668322 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:16.668394 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:16.668693 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 21:59:16.668796 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:59:16.669111 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.656952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33021 (local address 127.6.241.129:55173)
0620 21:59:16.657122 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:16.657165 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:16.657660 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:16.657949 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:16.657954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:16.658420 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:16.658768 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:16.658777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.660073 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.660074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:16.661625 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:16.661630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.662196 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.662197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:16.662320 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:16.662991 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:16.663004 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:16.664934 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:16.667213 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:16.667226 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:16.667229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:16.667622 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:16.667901 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:16.667902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:16.667903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:16.668045 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:16.668385 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:16.668392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:16.668544 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 21:59:16.668642 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 21:59:16.668804 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260620 21:59:16.670877 14831 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33021
I20260620 21:59:16.670974 14831 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:16.671159 14831 heartbeater.cc:511] Master 127.6.241.190:33021 requested a full tablet report, sending...
I20260620 21:59:16.671821 14627 ts_manager.cc:194] Registered new tserver with Master: f36f11e47daa42778564432d5bd92867 (127.6.241.129:43489)
I20260620 21:59:16.672801 14627 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:55173
WARNING: no policy specified for oryx/127.6.241.130@KRBTEST.COM; defaulting to no policy
I20260620 21:59:16.678349 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.671561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:41283 (local address 127.6.241.190:33021)
0620 21:59:16.671606 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:16.671608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:16.671729 (+ 121us) server_negotiation.cc:407] Connection header received
0620 21:59:16.671760 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:16.671761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:16.671779 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:16.671848 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:16.672222 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.672965 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.674177 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.674383 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.674445 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:16.675716 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:16.675718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:16.675718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:16.675726 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:16.677310 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.677728 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.677729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.677729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.677778 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.677931 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.677932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.677932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.678035 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:16.678080 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:16.678197 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 21:59:16.678258 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 21:59:16.678421 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.671506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33021 (local address 127.6.241.129:41283)
0620 21:59:16.671647 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:16.671657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:16.671724 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:16.671862 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:16.671863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:16.672062 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:16.672133 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:16.672135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.673056 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.673057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:16.674080 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:16.674084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.674471 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.674472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:16.674545 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:16.675105 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:16.675111 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:16.675564 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:16.677402 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:16.677404 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:16.677404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:16.677654 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:16.677834 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:16.677835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:16.677835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:16.677888 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:16.678081 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:16.678081 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:16.678123 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 21:59:16.678182 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:59:16.678271 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab.
Entry for principal oryx/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab.
I20260620 21:59:16.698436 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33021
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab
--principal=oryx/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:16.788969 14840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:16.789251 14840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:16.789286 14840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:16.790835 14840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:16.790896 14840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:16.790921 14840 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:59:16.790941 14840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:16.792742 14840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab
--principal=oryx/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.241.190:33021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:16.793051 14840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:16.793318 14840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:16.799986 14848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:16.799974 14846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:16.799998 14840 server_base.cc:1034] running on GCE node
W20260620 21:59:16.799974 14845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:16.800514 14840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:16.800798 14840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:16.802006 14840 hybrid_clock.cc:648] HybridClock initialized: now 1781992756801986 us; error 29 us; skew 500 ppm
Jun 20 21:59:16 dist-test-slave-gl2g krb5kdc[14583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:16.805272 14840 init.cc:377] Logged in from keytab as oryx/127.6.241.130@KRBTEST.COM (short username oryx)
I20260620 21:59:16.805876 14840 webserver.cc:460] Webserver started at http://127.6.241.130:42315/ using document root <none> and password file <none>
I20260620 21:59:16.806128 14840 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:16.806205 14840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:16.806360 14840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:16.807956 14840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "c09b7a8bef154751927bd6cdefe4a107"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "ef1ee9bbdb4e632b2210c740d8a72283"
server_key_iv: "943e457ed09fc0615f1e99d68a6af1d5"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.808437 14840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "c09b7a8bef154751927bd6cdefe4a107"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "ef1ee9bbdb4e632b2210c740d8a72283"
server_key_iv: "943e457ed09fc0615f1e99d68a6af1d5"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.810039 14840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:16.811013 14855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:16.811394 14840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:16.811534 14840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "c09b7a8bef154751927bd6cdefe4a107"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "ef1ee9bbdb4e632b2210c740d8a72283"
server_key_iv: "943e457ed09fc0615f1e99d68a6af1d5"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.811628 14840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:16.825134 14840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:16.827997 14840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:16.828219 14840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:16.828943 14840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:16.828999 14840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:16.829036 14840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:16.829059 14840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:59:16 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:16.837636 14840 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:34809
I20260620 21:59:16.837744 14970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:34809 every 8 connection(s)
I20260620 21:59:16.838064 14840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:16.840703 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.829449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:47945 (local address 127.6.241.190:33021)
0620 21:59:16.829524 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:16.829527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:16.829788 (+ 261us) server_negotiation.cc:407] Connection header received
0620 21:59:16.830336 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:16.830338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:16.830353 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:16.830439 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:16.831283 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.832386 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.833665 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.834061 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.834152 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:16.837462 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:16.837464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:16.837465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:16.837475 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:16.839275 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.839761 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.839762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.839762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.839822 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.840064 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.840064 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.840065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.840213 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:16.840269 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:16.840555 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 21:59:16.840629 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:59:16.841017 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.829447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33021 (local address 127.6.241.130:47945)
0620 21:59:16.829638 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:16.829684 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:16.830233 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:16.830551 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:16.830555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:16.830944 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:16.831170 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:16.831175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.832540 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.832541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:16.833574 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:16.833577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.834212 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.834213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:16.834400 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:16.835303 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:16.835315 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:16.837296 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:16.839400 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:16.839407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:16.839410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:16.839665 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:16.839886 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:16.839887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:16.839887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:16.840005 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:16.840269 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:16.840274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:16.840430 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 21:59:16.840523 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:59:16.840671 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260620 21:59:16.841645 14971 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33021
I20260620 21:59:16.841763 14971 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:16.841979 14971 heartbeater.cc:511] Master 127.6.241.190:33021 requested a full tablet report, sending...
I20260620 21:59:16.842568 14627 ts_manager.cc:194] Registered new tserver with Master: c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809)
I20260620 21:59:16.843295 14627 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:47945
I20260620 21:59:16.844460 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 14840
I20260620 21:59:16.844589 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:16.844844 7110 external_mini_cluster.cc:1428] Setting key c534c391f1644901083aed6af28d08a9
I20260620 21:59:16.850646 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.843140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:46039 (local address 127.6.241.190:33021)
0620 21:59:16.843191 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:16.843193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:16.843351 (+ 158us) server_negotiation.cc:407] Connection header received
0620 21:59:16.843455 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:16.843456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:16.843468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:16.843531 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:16.843915 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.844682 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.845784 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.846103 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.846254 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:16.847535 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:16.847537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:16.847537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:16.847546 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:16.849165 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.849631 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.849632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.849633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.849696 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:16.849915 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:16.849916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:16.849916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:16.850112 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:16.850217 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:16.850398 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 21:59:16.850497 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:59:16.850670 14977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.842979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33021 (local address 127.6.241.130:46039)
0620 21:59:16.843267 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:16.843279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:16.843363 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:16.843551 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:16.843552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:16.843743 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:16.843824 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:16.843826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.844995 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.844997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:16.845675 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:16.845677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:16.846267 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:16.846269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:16.846349 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:16.847011 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:16.847017 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:16.847395 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:16.849315 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:16.849317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:16.849318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:16.849531 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:16.849775 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:16.849776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:16.849777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:16.849847 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:16.850210 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:16.850211 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:16.850272 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 21:59:16.850368 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:59:16.850502 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for oryx/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab.
Entry for principal oryx/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab.
I20260620 21:59:16.880388 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33021
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab
--principal=oryx/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:16.976444 14980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:16.976760 14980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:16.976794 14980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:16.978425 14980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:16.978509 14980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:16.978538 14980 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:59:16.978562 14980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:16.980862 14980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab
--principal=oryx/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.241.190:33021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:16.981349 14980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:16.981734 14980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:16.988469 14980 server_base.cc:1034] running on GCE node
W20260620 21:59:16.988479 14986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:16.988481 14988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:16.988637 14985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:16.988935 14980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:16.989336 14980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:16.990643 14980 hybrid_clock.cc:648] HybridClock initialized: now 1781992756990567 us; error 37 us; skew 500 ppm
Jun 20 21:59:16 dist-test-slave-gl2g krb5kdc[14583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:16.993482 14980 init.cc:377] Logged in from keytab as oryx/127.6.241.131@KRBTEST.COM (short username oryx)
I20260620 21:59:16.993960 14980 webserver.cc:460] Webserver started at http://127.6.241.131:45685/ using document root <none> and password file <none>
I20260620 21:59:16.994186 14980 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:16.994236 14980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:16.994350 14980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:16.995647 14980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "9ee31c311603460d9f8c9f9040513102"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "d28a2ee051e7a3e8ff4aa3f00b116131"
server_key_iv: "96885165eeb542eab9feac6c93ae65eb"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.995980 14980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "9ee31c311603460d9f8c9f9040513102"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "d28a2ee051e7a3e8ff4aa3f00b116131"
server_key_iv: "96885165eeb542eab9feac6c93ae65eb"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.997617 14980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 21:59:16.998420 14995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:16.998673 14980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:16.998759 14980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "9ee31c311603460d9f8c9f9040513102"
format_stamp: "Formatted at 2026-06-20 21:59:16 on dist-test-slave-gl2g"
server_key: "d28a2ee051e7a3e8ff4aa3f00b116131"
server_key_iv: "96885165eeb542eab9feac6c93ae65eb"
server_key_version: "encryptionkey@0"
I20260620 21:59:16.998823 14980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:17.021260 14980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:17.023679 14980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:17.023802 14980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:17.024307 14980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:17.024343 14980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.024367 14980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:17.024380 14980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:17.031751 14980 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:38343
I20260620 21:59:17.031840 15110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:38343 every 8 connection(s)
I20260620 21:59:17.032164 14980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:17.035271 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.024954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:49413 (local address 127.6.241.190:33021)
0620 21:59:17.025025 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.025028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.025244 (+ 216us) server_negotiation.cc:407] Connection header received
0620 21:59:17.025922 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.025924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.025939 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.026029 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:17.026950 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.027841 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.029127 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.029520 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.029606 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.032289 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:17.032291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:17.032291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:17.032302 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:17.033936 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.034339 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.034340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.034341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.034390 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.034641 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.034642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.034643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.034771 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:17.034868 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.035122 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.035205 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 21:59:17.035588 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.024933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33021 (local address 127.6.241.131:49413)
0620 21:59:17.025108 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.025159 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.025794 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.026149 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.026153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.026580 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:17.026847 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.026851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.027983 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.027984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.029022 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.029025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.029639 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.029641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.029811 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.030441 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:17.030450 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:17.032155 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:17.034018 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:17.034025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:17.034026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:17.034271 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:17.034442 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:17.034442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:17.034443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:17.034559 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:17.034878 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:17.034883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:17.035012 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.035103 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.035348 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260620 21:59:17.036187 15111 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33021
I20260620 21:59:17.036303 15111 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:17.036499 15111 heartbeater.cc:511] Master 127.6.241.190:33021 requested a full tablet report, sending...
I20260620 21:59:17.036816 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 14980
I20260620 21:59:17.036916 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:17.036937 14627 ts_manager.cc:194] Registered new tserver with Master: 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343)
I20260620 21:59:17.037081 7110 external_mini_cluster.cc:1428] Setting key f8a004ca7bcd89c2d56089da213b4b1b
I20260620 21:59:17.037456 14627 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:49413
I20260620 21:59:17.037979 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:17.044250 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.037727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33021 (local address 127.6.241.131:41889)
0620 21:59:17.037859 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.037870 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.037951 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.038110 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.038112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.038335 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:17.038398 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.038401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.039150 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.039151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.039703 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.039706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.040101 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.040102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.040171 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.040728 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:17.040733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:17.041092 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:17.043134 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:17.043135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:17.043135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:17.043324 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:17.043568 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:17.043569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:17.043569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:17.043628 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:17.043888 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:17.043889 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:17.043951 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.044028 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.044144 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":84,"threads_started":1}
I20260620 21:59:17.044245 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.037764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:41889 (local address 127.6.241.190:33021)
0620 21:59:17.037964 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.037966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.037975 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:17.037998 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.037998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.038009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.038061 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:17.038473 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.039048 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.039793 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.040017 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.040093 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.041196 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:17.041197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:17.041198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:17.041206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:17.043058 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.043427 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.043429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.043429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.043494 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.043687 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.043688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.043689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.043819 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:17.043900 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.044054 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.044142 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:59:17.062402 14692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:17.055925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.6.241.190:33021)
0620 21:59:17.055990 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.055992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.056001 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:17.056028 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.056029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.056039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.056121 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:17.056861 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.057358 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.058152 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.058352 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.058421 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.062264 (+ 3843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44346: BlockingRecv error: recv got EOF from 127.0.0.1:44346 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:17.071182 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.063342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.6.241.190:33021)
0620 21:59:17.063420 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.063422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.063518 (+ 96us) server_negotiation.cc:407] Connection header received
0620 21:59:17.063566 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.063568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.063581 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.063666 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:17.064153 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.064690 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.065494 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.065691 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.065761 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.068322 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:17.068324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:17.068324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:17.068333 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:17.070058 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.070450 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.070452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.070452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.070522 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.070746 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.070747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.070747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.070867 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:17.070946 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.071035 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.071104 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:59:17.072985 14627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:17.074726 14627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:17.081596 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.077389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.6.241.129:43489)
0620 21:59:17.077571 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.077576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.077589 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:59:17.077633 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.077635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.077661 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.077750 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:17.077953 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.078908 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.080460 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.081190 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.081259 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.081331 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.081450 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.081488 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260620 21:59:17.081925 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.077377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.6.241.130:34809)
0620 21:59:17.077559 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.077564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.077962 (+ 398us) server_negotiation.cc:407] Connection header received
0620 21:59:17.078006 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.078009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.078025 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.078101 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:17.078196 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.079267 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.080361 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.081125 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.081206 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.081291 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.081784 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.081828 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260620 21:59:17.082556 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.077050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:43489 (local address 127.0.0.1:53208)
0620 21:59:17.077231 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.077271 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.077338 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.077765 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.077768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.077770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:17.077861 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.077864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.079029 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.079030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.080397 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.080398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.081258 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.081259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.081344 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.081348 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.081428 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.081498 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":33,"thread_start_us":69,"threads_started":1}
I20260620 21:59:17.082574 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.077269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:34809 (local address 127.0.0.1:45982)
0620 21:59:17.077841 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.077851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.077901 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.078099 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.078100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.078101 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:17.078136 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.078137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.079335 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.079336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.080273 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.080276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.081221 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.081238 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.081330 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.081348 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.081428 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.081498 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":90,"threads_started":1}
I20260620 21:59:17.082942 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.077050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:38343 (local address 127.0.0.1:35142)
0620 21:59:17.077446 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.077453 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.077505 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.077824 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.077825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.077827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:17.077889 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.077891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.079255 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.079256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.080377 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.080380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.082523 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.082524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.082618 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.082623 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.082814 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.082863 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":5,"thread_start_us":72,"threads_started":1}
I20260620 21:59:17.083215 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.077187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.6.241.131:38343)
0620 21:59:17.077583 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.077587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.077596 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:17.077633 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.077635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.077651 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.077754 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:17.077971 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.079172 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.081372 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.082345 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.082411 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.082485 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.083060 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.083110 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":93,"threads_started":1}
I20260620 21:59:17.083935 15045 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac3df9d46f142ce9cd15483876d4793 (DEFAULT_TABLE table=test-table [id=97a5dd095cbb4272bc5a66cf00d4c53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.084126 14765 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac3df9d46f142ce9cd15483876d4793 (DEFAULT_TABLE table=test-table [id=97a5dd095cbb4272bc5a66cf00d4c53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.084417 15045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac3df9d46f142ce9cd15483876d4793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.084306 14905 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac3df9d46f142ce9cd15483876d4793 (DEFAULT_TABLE table=test-table [id=97a5dd095cbb4272bc5a66cf00d4c53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.084585 14765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac3df9d46f142ce9cd15483876d4793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.084619 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac3df9d46f142ce9cd15483876d4793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.088157 15139 tablet_bootstrap.cc:492] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: Bootstrap starting.
I20260620 21:59:17.088617 15140 tablet_bootstrap.cc:492] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: Bootstrap starting.
I20260620 21:59:17.089046 15139 tablet_bootstrap.cc:654] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.089243 15140 tablet_bootstrap.cc:654] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.089345 15141 tablet_bootstrap.cc:492] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: Bootstrap starting.
I20260620 21:59:17.089545 15139 log.cc:824] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:17.089565 15140 log.cc:824] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:17.090008 15141 tablet_bootstrap.cc:654] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.090320 15139 tablet_bootstrap.cc:492] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: No bootstrap required, opened a new log
I20260620 21:59:17.090344 15141 log.cc:824] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:17.090423 15139 ts_tablet_manager.cc:1397] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:17.090710 15140 tablet_bootstrap.cc:492] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: No bootstrap required, opened a new log
I20260620 21:59:17.090785 15140 ts_tablet_manager.cc:1397] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:17.091069 15141 tablet_bootstrap.cc:492] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: No bootstrap required, opened a new log
I20260620 21:59:17.091149 15141 ts_tablet_manager.cc:1397] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:17.092496 15141 raft_consensus.cc:348] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } }
I20260620 21:59:17.092667 15141 raft_consensus.cc:374] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.092553 15139 raft_consensus.cc:348] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } }
I20260620 21:59:17.092712 15141 raft_consensus.cc:729] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09b7a8bef154751927bd6cdefe4a107, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.092727 15139 raft_consensus.cc:374] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.092763 15139 raft_consensus.cc:729] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36f11e47daa42778564432d5bd92867, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.092783 15140 raft_consensus.cc:348] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } }
I20260620 21:59:17.092857 15141 consensus_queue.cc:260] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } }
I20260620 21:59:17.092929 15140 raft_consensus.cc:374] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.092890 15139 consensus_queue.cc:260] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } }
I20260620 21:59:17.092971 15140 raft_consensus.cc:729] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee31c311603460d9f8c9f9040513102, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.093103 15140 consensus_queue.cc:260] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } }
I20260620 21:59:17.093147 15141 ts_tablet_manager.cc:1428] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:17.093163 15139 ts_tablet_manager.cc:1428] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:17.093264 14971 heartbeater.cc:503] Master 127.6.241.190:33021 was elected leader, sending a full tablet report...
I20260620 21:59:17.093276 14831 heartbeater.cc:503] Master 127.6.241.190:33021 was elected leader, sending a full tablet report...
I20260620 21:59:17.093626 15140 ts_tablet_manager.cc:1428] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:17.093740 15111 heartbeater.cc:503] Master 127.6.241.190:33021 was elected leader, sending a full tablet report...
W20260620 21:59:17.165310 14832 tablet.cc:2369] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:17.174151 15146 raft_consensus.cc:482] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:17.174325 15146 raft_consensus.cc:504] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } }
I20260620 21:59:17.174741 15146 leader_election.cc:288] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343), c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809)
I20260620 21:59:17.178412 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.175224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:39565 (local address 127.6.241.130:34809)
0620 21:59:17.175265 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.175267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.175297 (+ 30us) server_negotiation.cc:407] Connection header received
0620 21:59:17.175369 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.175370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.175377 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.175415 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:17.176152 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.176600 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.177677 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.178033 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.178112 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.178120 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.178294 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.178343 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 21:59:17.178412 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.175039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:35259 (local address 127.6.241.131:38343)
0620 21:59:17.175109 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.175112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.175164 (+ 52us) server_negotiation.cc:407] Connection header received
0620 21:59:17.175277 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.175278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.175289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.175347 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:17.175591 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.176060 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.177211 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.177571 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.177622 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.177629 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.178308 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.178342 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:59:17.178427 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.175136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:34809 (local address 127.6.241.129:39565)
0620 21:59:17.175243 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.175252 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.175308 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.175422 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.175422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.175423 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:17.175519 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.175522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.176682 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.176683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.177567 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.177569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.178113 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.178114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.178193 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.178197 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.178256 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.178302 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":4,"thread_start_us":57,"threads_started":1}
I20260620 21:59:17.178438 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.174933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:38343 (local address 127.6.241.129:35259)
0620 21:59:17.175074 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.175087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.175182 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.175420 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.175421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.175423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:17.175518 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.175521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.176123 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.176124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.177123 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.177125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.178151 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.178152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.178213 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.178216 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.178271 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.178302 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":24,"thread_start_us":75,"threads_started":1}
I20260620 21:59:17.178794 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac3df9d46f142ce9cd15483876d4793" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09b7a8bef154751927bd6cdefe4a107" is_pre_election: true
I20260620 21:59:17.178968 14925 raft_consensus.cc:2454] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 0.
I20260620 21:59:17.179177 14719 leader_election.cc:302] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c09b7a8bef154751927bd6cdefe4a107, f36f11e47daa42778564432d5bd92867; no voters:
I20260620 21:59:17.179306 15146 raft_consensus.cc:2781] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:17.179368 15146 raft_consensus.cc:482] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:17.179392 15146 raft_consensus.cc:3037] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.179518 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac3df9d46f142ce9cd15483876d4793" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee31c311603460d9f8c9f9040513102" is_pre_election: true
I20260620 21:59:17.179661 15065 raft_consensus.cc:2454] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 0.
I20260620 21:59:17.180202 15146 raft_consensus.cc:504] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } }
I20260620 21:59:17.180336 15146 leader_election.cc:288] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 election: Requested vote from peers 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343), c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809)
I20260620 21:59:17.180467 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac3df9d46f142ce9cd15483876d4793" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee31c311603460d9f8c9f9040513102"
I20260620 21:59:17.180498 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac3df9d46f142ce9cd15483876d4793" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09b7a8bef154751927bd6cdefe4a107"
I20260620 21:59:17.180548 15065 raft_consensus.cc:3037] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.180562 14925 raft_consensus.cc:3037] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.181430 15065 raft_consensus.cc:2454] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 1.
I20260620 21:59:17.181430 14925 raft_consensus.cc:2454] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 1.
I20260620 21:59:17.181600 14719 leader_election.cc:302] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c09b7a8bef154751927bd6cdefe4a107, f36f11e47daa42778564432d5bd92867; no voters:
I20260620 21:59:17.181694 15146 raft_consensus.cc:2781] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:17.181838 15146 raft_consensus.cc:686] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 1 LEADER]: Becoming Leader. State: Replica: f36f11e47daa42778564432d5bd92867, State: Running, Role: LEADER
I20260620 21:59:17.181936 15146 consensus_queue.cc:237] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } }
I20260620 21:59:17.182765 14627 catalog_manager.cc:5495] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 reported cstate change: term changed from 0 to 1, leader changed from <none> to f36f11e47daa42778564432d5bd92867 (127.6.241.129). New cstate: current_term: 1 leader_uuid: "f36f11e47daa42778564432d5bd92867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } health_report { overall_health: UNKNOWN } } }
I20260620 21:59:17.197698 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.195467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.6.241.129:43489)
0620 21:59:17.195532 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.195535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.195544 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:17.195587 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.195588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.195607 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.195677 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:17.195800 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.196347 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.197044 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.197226 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.197275 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.197424 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.197535 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.197599 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.197615 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.197635 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 21:59:17.199656 15065 raft_consensus.cc:1261] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Refusing update from remote peer f36f11e47daa42778564432d5bd92867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.199656 14925 raft_consensus.cc:1261] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Refusing update from remote peer f36f11e47daa42778564432d5bd92867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.200055 15146 consensus_queue.cc:1035] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.200199 15150 consensus_queue.cc:1035] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.203339 15155 mvcc.cc:205] Tried to move back new op lower bound from 7299042333486698496 to 7299042333417873408. Current Snapshot: MvccSnapshot[applied={T|T < 7299042333486698496 or (T in {7299042333486698496})}]
I20260620 21:59:17.205060 14627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:17.205160 14627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:17.205888 14627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: Sending DeleteTablet for 3 replicas of tablet 8ac3df9d46f142ce9cd15483876d4793
I20260620 21:59:17.206192 15045 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac3df9d46f142ce9cd15483876d4793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:35142
I20260620 21:59:17.206218 14765 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac3df9d46f142ce9cd15483876d4793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:53208
I20260620 21:59:17.206458 15164 tablet_replica.cc:330] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: stopping tablet replica
I20260620 21:59:17.206431 14905 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac3df9d46f142ce9cd15483876d4793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:45982
I20260620 21:59:17.206573 15164 raft_consensus.cc:2229] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:17.206743 15164 raft_consensus.cc:2258] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.206964 15163 tablet_replica.cc:330] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: stopping tablet replica
I20260620 21:59:17.207005 15165 tablet_replica.cc:330] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: stopping tablet replica
I20260620 21:59:17.207072 15163 raft_consensus.cc:2229] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:17.207022 14627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:59:17.207176 15163 raft_consensus.cc:2258] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:59:17.207185 14627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:17.207361 15165 raft_consensus.cc:2229] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:17.207546 15165 raft_consensus.cc:2258] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.207628 15163 ts_tablet_manager.cc:1905] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.208077 15165 ts_tablet_manager.cc:1905] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.208642 14765 tablet_service.cc:1467] Processing CreateTablet for tablet 8f98bc7c649847979aca89cdc62470be (DEFAULT_TABLE table=test-table [id=5e795a3ee0e8425092a22323a9449f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.208767 14765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f98bc7c649847979aca89cdc62470be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.209936 15163 ts_tablet_manager.cc:1918] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:17.209939 15165 ts_tablet_manager.cc:1918] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:17.210016 15163 log.cc:1196] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/8ac3df9d46f142ce9cd15483876d4793
I20260620 21:59:17.210177 15165 log.cc:1196] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/8ac3df9d46f142ce9cd15483876d4793
I20260620 21:59:17.210261 15139 tablet_bootstrap.cc:492] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: Bootstrap starting.
I20260620 21:59:17.210331 15163 ts_tablet_manager.cc:1939] T 8ac3df9d46f142ce9cd15483876d4793 P 9ee31c311603460d9f8c9f9040513102: Deleting consensus metadata
I20260620 21:59:17.210639 15165 ts_tablet_manager.cc:1939] T 8ac3df9d46f142ce9cd15483876d4793 P c09b7a8bef154751927bd6cdefe4a107: Deleting consensus metadata
I20260620 21:59:17.210726 15139 tablet_bootstrap.cc:654] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.211249 14615 catalog_manager.cc:4841] TS c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809): tablet 8ac3df9d46f142ce9cd15483876d4793 (table test-table [id=97a5dd095cbb4272bc5a66cf00d4c53c]) successfully deleted
I20260620 21:59:17.211315 14905 tablet_service.cc:1467] Processing CreateTablet for tablet 8f98bc7c649847979aca89cdc62470be (DEFAULT_TABLE table=test-table [id=5e795a3ee0e8425092a22323a9449f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.211349 14615 catalog_manager.cc:4841] TS 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343): tablet 8ac3df9d46f142ce9cd15483876d4793 (table test-table [id=97a5dd095cbb4272bc5a66cf00d4c53c]) successfully deleted
I20260620 21:59:17.211398 15139 tablet_bootstrap.cc:492] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: No bootstrap required, opened a new log
I20260620 21:59:17.211409 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f98bc7c649847979aca89cdc62470be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.211441 15139 ts_tablet_manager.cc:1397] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:17.211504 15045 tablet_service.cc:1467] Processing CreateTablet for tablet 8f98bc7c649847979aca89cdc62470be (DEFAULT_TABLE table=test-table [id=5e795a3ee0e8425092a22323a9449f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.211591 15045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f98bc7c649847979aca89cdc62470be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.211577 15139 raft_consensus.cc:348] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.211655 15139 raft_consensus.cc:374] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.211678 15139 raft_consensus.cc:729] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36f11e47daa42778564432d5bd92867, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.211721 15139 consensus_queue.cc:260] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.211803 15139 ts_tablet_manager.cc:1428] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.212620 15141 tablet_bootstrap.cc:492] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: Bootstrap starting.
I20260620 21:59:17.212939 15140 tablet_bootstrap.cc:492] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: Bootstrap starting.
I20260620 21:59:17.213016 15141 tablet_bootstrap.cc:654] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.213315 15140 tablet_bootstrap.cc:654] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.213353 15164 ts_tablet_manager.cc:1905] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.213582 15141 tablet_bootstrap.cc:492] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: No bootstrap required, opened a new log
I20260620 21:59:17.213622 15141 ts_tablet_manager.cc:1397] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:17.213765 15141 raft_consensus.cc:348] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.213828 15141 raft_consensus.cc:374] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.213850 15141 raft_consensus.cc:729] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09b7a8bef154751927bd6cdefe4a107, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.213896 15141 consensus_queue.cc:260] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.213981 15141 ts_tablet_manager.cc:1428] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.215288 15164 ts_tablet_manager.cc:1918] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:17.215374 15164 log.cc:1196] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/8ac3df9d46f142ce9cd15483876d4793
I20260620 21:59:17.215518 15140 tablet_bootstrap.cc:492] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: No bootstrap required, opened a new log
I20260620 21:59:17.215586 15140 ts_tablet_manager.cc:1397] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 21:59:17.215662 15164 ts_tablet_manager.cc:1939] T 8ac3df9d46f142ce9cd15483876d4793 P f36f11e47daa42778564432d5bd92867: Deleting consensus metadata
I20260620 21:59:17.215754 15140 raft_consensus.cc:348] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.215848 15140 raft_consensus.cc:374] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.215876 15140 raft_consensus.cc:729] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee31c311603460d9f8c9f9040513102, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.215919 15140 consensus_queue.cc:260] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.216005 15140 ts_tablet_manager.cc:1428] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.216055 14612 catalog_manager.cc:4841] TS f36f11e47daa42778564432d5bd92867 (127.6.241.129:43489): tablet 8ac3df9d46f142ce9cd15483876d4793 (table test-table [id=97a5dd095cbb4272bc5a66cf00d4c53c]) successfully deleted
I20260620 21:59:17.228279 15158 raft_consensus.cc:482] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:17.228404 15158 raft_consensus.cc:504] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.228605 15158 leader_election.cc:288] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809), 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343)
I20260620 21:59:17.228811 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f98bc7c649847979aca89cdc62470be" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee31c311603460d9f8c9f9040513102" is_pre_election: true
I20260620 21:59:17.228814 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f98bc7c649847979aca89cdc62470be" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09b7a8bef154751927bd6cdefe4a107" is_pre_election: true
I20260620 21:59:17.228919 14925 raft_consensus.cc:2454] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 0.
I20260620 21:59:17.228937 15065 raft_consensus.cc:2454] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 0.
I20260620 21:59:17.229106 14719 leader_election.cc:302] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c09b7a8bef154751927bd6cdefe4a107, f36f11e47daa42778564432d5bd92867; no voters:
I20260620 21:59:17.229209 15158 raft_consensus.cc:2781] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:17.229276 15158 raft_consensus.cc:482] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:17.229305 15158 raft_consensus.cc:3037] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.230140 15158 raft_consensus.cc:504] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.230278 15158 leader_election.cc:288] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 election: Requested vote from peers c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809), 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343)
I20260620 21:59:17.230479 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f98bc7c649847979aca89cdc62470be" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee31c311603460d9f8c9f9040513102"
I20260620 21:59:17.230469 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f98bc7c649847979aca89cdc62470be" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09b7a8bef154751927bd6cdefe4a107"
I20260620 21:59:17.230551 15065 raft_consensus.cc:3037] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.230551 14925 raft_consensus.cc:3037] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.231313 15065 raft_consensus.cc:2454] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 1.
I20260620 21:59:17.231318 14925 raft_consensus.cc:2454] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 1.
I20260620 21:59:17.231498 14719 leader_election.cc:302] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ee31c311603460d9f8c9f9040513102, f36f11e47daa42778564432d5bd92867; no voters:
I20260620 21:59:17.231597 15158 raft_consensus.cc:2781] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:17.231650 15158 raft_consensus.cc:686] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 1 LEADER]: Becoming Leader. State: Replica: f36f11e47daa42778564432d5bd92867, State: Running, Role: LEADER
I20260620 21:59:17.231705 15158 consensus_queue.cc:237] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.232287 14627 catalog_manager.cc:5495] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 reported cstate change: term changed from 0 to 1, leader changed from <none> to f36f11e47daa42778564432d5bd92867 (127.6.241.129). New cstate: current_term: 1 leader_uuid: "f36f11e47daa42778564432d5bd92867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } health_report { overall_health: UNKNOWN } } }
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992756, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.190@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:17.267272 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.259508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.6.241.190:33021)
0620 21:59:17.259576 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.259579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.259721 (+ 142us) server_negotiation.cc:407] Connection header received
0620 21:59:17.259803 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.259803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.259813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.259856 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:17.260477 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.260965 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.261987 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.262167 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.262221 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.264591 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:17.264592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:17.264592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:17.264601 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:17.265987 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.266394 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.266396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.266396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.266450 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.266684 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.266684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.266685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.266801 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:17.266878 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.267037 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.267198 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":36}
I20260620 21:59:17.267272 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.259451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33021 (local address 127.0.0.1:44366)
0620 21:59:17.259638 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.259651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.259733 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.259886 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.259887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.260333 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:17.260393 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.260396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.261046 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.261047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.261905 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.261908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.262241 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.262242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.262303 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.262938 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:17.262944 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:17.264494 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:17.266091 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:17.266100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:17.266103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:17.266305 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:17.266512 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:17.266513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:17.266514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:17.266570 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:17.266867 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:17.266875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:17.266923 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.267002 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.267150 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":5,"thread_start_us":107,"threads_started":1}
I20260620 21:59:17.269491 14627 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:44366:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:59:17.269703 14627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:17.271173 14905 tablet_service.cc:1467] Processing CreateTablet for tablet ca30a62e1bf14bd58822bc105dbf5e60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10dbc635c40041b3bc0368a26d60636a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:59:17.271188 15045 tablet_service.cc:1467] Processing CreateTablet for tablet ca30a62e1bf14bd58822bc105dbf5e60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10dbc635c40041b3bc0368a26d60636a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:59:17.271183 14765 tablet_service.cc:1467] Processing CreateTablet for tablet ca30a62e1bf14bd58822bc105dbf5e60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10dbc635c40041b3bc0368a26d60636a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:59:17.271337 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca30a62e1bf14bd58822bc105dbf5e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.271346 15045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca30a62e1bf14bd58822bc105dbf5e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.271346 14765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca30a62e1bf14bd58822bc105dbf5e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.272857 15140 tablet_bootstrap.cc:492] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: Bootstrap starting.
I20260620 21:59:17.272881 15141 tablet_bootstrap.cc:492] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: Bootstrap starting.
I20260620 21:59:17.273232 15140 tablet_bootstrap.cc:654] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.273324 15141 tablet_bootstrap.cc:654] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.273655 15139 tablet_bootstrap.cc:492] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: Bootstrap starting.
I20260620 21:59:17.273826 15140 tablet_bootstrap.cc:492] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: No bootstrap required, opened a new log
I20260620 21:59:17.273859 15140 ts_tablet_manager.cc:1397] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:17.273927 15139 tablet_bootstrap.cc:654] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.273969 15140 raft_consensus.cc:348] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.274024 15140 raft_consensus.cc:374] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.274039 15140 raft_consensus.cc:729] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee31c311603460d9f8c9f9040513102, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.274098 15140 consensus_queue.cc:260] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.274196 15140 ts_tablet_manager.cc:1428] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.274358 15147 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.274458 15147 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:59:17.274526 15139 tablet_bootstrap.cc:492] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: No bootstrap required, opened a new log
I20260620 21:59:17.274575 15139 ts_tablet_manager.cc:1397] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:17.274752 15139 raft_consensus.cc:348] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.274832 15139 raft_consensus.cc:374] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.274853 15139 raft_consensus.cc:729] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36f11e47daa42778564432d5bd92867, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.274896 15139 consensus_queue.cc:260] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.274930 15141 tablet_bootstrap.cc:492] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: No bootstrap required, opened a new log
I20260620 21:59:17.274974 15141 ts_tablet_manager.cc:1397] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:17.274988 15139 ts_tablet_manager.cc:1428] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.275089 15158 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.275138 15158 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:59:17.275123 15141 raft_consensus.cc:348] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.275187 15141 raft_consensus.cc:374] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.275214 15141 raft_consensus.cc:729] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09b7a8bef154751927bd6cdefe4a107, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.275257 15141 consensus_queue.cc:260] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.275349 15141 ts_tablet_manager.cc:1428] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.275538 15145 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.275626 15145 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 21:59:17.282742 15112 tablet.cc:2369] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:59:17.338766 14972 tablet.cc:2369] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:17.497941 15147 raft_consensus.cc:482] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:17.498086 15147 raft_consensus.cc:504] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.498495 15147 leader_election.cc:288] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f36f11e47daa42778564432d5bd92867 (127.6.241.129:43489), c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809)
I20260620 21:59:17.501756 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.498784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:57825 (local address 127.6.241.130:34809)
0620 21:59:17.498829 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.498831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.498939 (+ 108us) server_negotiation.cc:407] Connection header received
0620 21:59:17.498974 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.498975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.498984 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.499041 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:17.499246 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.499994 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.501036 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.501396 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.501484 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.501492 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.501644 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.501687 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:59:17.501750 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.498752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:54685 (local address 127.6.241.129:43489)
0620 21:59:17.498801 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.498804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.498883 (+ 79us) server_negotiation.cc:407] Connection header received
0620 21:59:17.498958 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.498960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.498973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.499041 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:17.499247 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.500007 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.501015 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.501385 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.501467 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.501475 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.501631 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.501671 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 21:59:17.501775 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.498717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:43489 (local address 127.6.241.131:54685)
0620 21:59:17.498789 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.498801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.498881 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.499056 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.499056 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.499058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:17.499159 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.499161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.500072 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.500073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.500915 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.500918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.501464 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.501465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.501544 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.501547 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.501623 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.501691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":23}
I20260620 21:59:17.501802 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.498735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:34809 (local address 127.6.241.131:57825)
0620 21:59:17.498865 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.498875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.498930 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.499055 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.499056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.499058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:17.499160 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.499161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.500119 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.500120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.500950 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.500952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.501483 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.501484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.501555 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.501564 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.501634 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.501707 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":6,"thread_start_us":56,"threads_started":1}
I20260620 21:59:17.502061 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca30a62e1bf14bd58822bc105dbf5e60" candidate_uuid: "9ee31c311603460d9f8c9f9040513102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09b7a8bef154751927bd6cdefe4a107" is_pre_election: true
I20260620 21:59:17.502166 14925 raft_consensus.cc:2454] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ee31c311603460d9f8c9f9040513102 in term 0.
I20260620 21:59:17.502092 14785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca30a62e1bf14bd58822bc105dbf5e60" candidate_uuid: "9ee31c311603460d9f8c9f9040513102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36f11e47daa42778564432d5bd92867" is_pre_election: true
I20260620 21:59:17.502228 14785 raft_consensus.cc:2454] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ee31c311603460d9f8c9f9040513102 in term 0.
I20260620 21:59:17.502324 14999 leader_election.cc:302] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ee31c311603460d9f8c9f9040513102, c09b7a8bef154751927bd6cdefe4a107; no voters:
I20260620 21:59:17.502466 15147 raft_consensus.cc:2781] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:17.502512 15147 raft_consensus.cc:482] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:17.502544 15147 raft_consensus.cc:3037] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.503288 15147 raft_consensus.cc:504] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.503438 15147 leader_election.cc:288] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [CANDIDATE]: Term 1 election: Requested vote from peers f36f11e47daa42778564432d5bd92867 (127.6.241.129:43489), c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809)
I20260620 21:59:17.503650 14785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca30a62e1bf14bd58822bc105dbf5e60" candidate_uuid: "9ee31c311603460d9f8c9f9040513102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36f11e47daa42778564432d5bd92867"
I20260620 21:59:17.503743 14785 raft_consensus.cc:3037] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.503698 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca30a62e1bf14bd58822bc105dbf5e60" candidate_uuid: "9ee31c311603460d9f8c9f9040513102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09b7a8bef154751927bd6cdefe4a107"
I20260620 21:59:17.503789 14925 raft_consensus.cc:3037] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.504383 14925 raft_consensus.cc:2454] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ee31c311603460d9f8c9f9040513102 in term 1.
I20260620 21:59:17.504534 14785 raft_consensus.cc:2454] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ee31c311603460d9f8c9f9040513102 in term 1.
I20260620 21:59:17.504541 14999 leader_election.cc:302] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ee31c311603460d9f8c9f9040513102, c09b7a8bef154751927bd6cdefe4a107; no voters:
I20260620 21:59:17.504647 15147 raft_consensus.cc:2781] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:17.504799 15147 raft_consensus.cc:686] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [term 1 LEADER]: Becoming Leader. State: Replica: 9ee31c311603460d9f8c9f9040513102, State: Running, Role: LEADER
I20260620 21:59:17.504881 15147 consensus_queue.cc:237] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.505409 15180 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: TxnStatusTablet state changed. Reason: New leader 9ee31c311603460d9f8c9f9040513102. Latest consensus state: current_term: 1 leader_uuid: "9ee31c311603460d9f8c9f9040513102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.505499 15180 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:59:17.505676 15182 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:59:17.505636 14627 catalog_manager.cc:5495] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ee31c311603460d9f8c9f9040513102 (127.6.241.131). New cstate: current_term: 1 leader_uuid: "9ee31c311603460d9f8c9f9040513102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } health_report { overall_health: HEALTHY } } }
I20260620 21:59:17.505800 15182 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:59:17.560773 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.558271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:38343 (local address 127.0.0.1:35146)
0620 21:59:17.558351 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.558362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.558413 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.558623 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.558624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.558625 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:59:17.558680 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.558682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.559309 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.559310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.559887 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.559889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.560246 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.560247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.560318 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.560333 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.560589 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.560598 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.560654 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.560705 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 21:59:17.560773 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.558344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.6.241.131:38343)
0620 21:59:17.558394 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.558396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.558406 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:17.558498 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.558499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.558518 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.558596 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:17.558746 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.559223 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.559969 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.560166 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.560236 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.560420 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.560524 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.560606 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.560660 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.560681 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 21:59:17.561920 14925 raft_consensus.cc:1261] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Refusing update from remote peer 9ee31c311603460d9f8c9f9040513102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.562072 14785 raft_consensus.cc:1261] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Refusing update from remote peer 9ee31c311603460d9f8c9f9040513102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.562389 15180 consensus_queue.cc:1035] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.562561 15147 consensus_queue.cc:1035] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.563231 15145 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: TxnStatusTablet state changed. Reason: New leader 9ee31c311603460d9f8c9f9040513102. Latest consensus state: current_term: 1 leader_uuid: "9ee31c311603460d9f8c9f9040513102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.563347 15145 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:59:17.563374 15158 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: TxnStatusTablet state changed. Reason: New leader 9ee31c311603460d9f8c9f9040513102. Latest consensus state: current_term: 1 leader_uuid: "9ee31c311603460d9f8c9f9040513102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.563476 15158 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:59:17.564391 15147 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ee31c311603460d9f8c9f9040513102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.564505 15147 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:59:17.564477 15145 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ee31c311603460d9f8c9f9040513102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.564565 15145 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P c09b7a8bef154751927bd6cdefe4a107: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:59:17.564564 15154 mvcc.cc:205] Tried to move back new op lower bound from 7299042334971805696 to 7299042334740484096. Current Snapshot: MvccSnapshot[applied={T|T < 7299042334971805696}]
I20260620 21:59:17.564414 15180 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ee31c311603460d9f8c9f9040513102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.564759 15180 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P 9ee31c311603460d9f8c9f9040513102: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:59:17.565389 15158 tablet_replica.cc:439] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ee31c311603460d9f8c9f9040513102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } } }
I20260620 21:59:17.565500 15158 tablet_replica.cc:442] T ca30a62e1bf14bd58822bc105dbf5e60 P f36f11e47daa42778564432d5bd92867: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:59:17.565397 15153 mvcc.cc:205] Tried to move back new op lower bound from 7299042334971805696 to 7299042334740484096. Current Snapshot: MvccSnapshot[applied={T|T < 7299042334971805696}]
I20260620 21:59:17.571200 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.568250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:51269 (local address 127.6.241.131:38343)
0620 21:59:17.568299 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.568301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.568430 (+ 129us) server_negotiation.cc:407] Connection header received
0620 21:59:17.568505 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.568506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.568535 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.568580 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:17.568755 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.569489 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.570387 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.570702 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.570769 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.570955 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.570991 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.571041 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.571096 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.571117 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 21:59:17.571244 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.568190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:38343 (local address 127.6.241.129:51269)
0620 21:59:17.568357 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.568367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.568445 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.568626 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.568627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.568629 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:59:17.568683 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.568685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.569569 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.569570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.570293 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.570296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.570780 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.570782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.570842 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.570881 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.571046 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.571047 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.571094 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.571146 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260620 21:59:17.573987 15065 raft_consensus.cc:1261] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Refusing update from remote peer f36f11e47daa42778564432d5bd92867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.574005 14925 raft_consensus.cc:1261] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Refusing update from remote peer f36f11e47daa42778564432d5bd92867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.574241 15158 consensus_queue.cc:1035] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.574388 15146 consensus_queue.cc:1035] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.584381 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.581487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:33163 (local address 127.6.241.129:43489)
0620 21:59:17.581529 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.581532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.581658 (+ 126us) server_negotiation.cc:407] Connection header received
0620 21:59:17.581726 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.581727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.581738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.581792 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:17.581959 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.582534 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.583584 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.583794 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.583876 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.584068 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.584107 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.584154 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.584264 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.584290 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 21:59:17.584424 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.581427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:43489 (local address 127.6.241.131:33163)
0620 21:59:17.581589 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.581600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.581675 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.581814 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.581815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.581817 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:59:17.581894 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.581896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.582634 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.582635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.583501 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.583504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.583896 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.583897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.583961 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.583977 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.584176 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.584178 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.584254 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.584315 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260620 21:59:17.593117 14627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:17.593255 14627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:17.593941 14627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: Sending DeleteTablet for 3 replicas of tablet 8f98bc7c649847979aca89cdc62470be
I20260620 21:59:17.594242 14905 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f98bc7c649847979aca89cdc62470be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:45982
I20260620 21:59:17.594358 14765 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f98bc7c649847979aca89cdc62470be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:53208
I20260620 21:59:17.594355 15045 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f98bc7c649847979aca89cdc62470be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:35142
I20260620 21:59:17.594456 15165 tablet_replica.cc:330] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: stopping tablet replica
I20260620 21:59:17.594466 15163 tablet_replica.cc:330] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: stopping tablet replica
I20260620 21:59:17.594468 15164 tablet_replica.cc:330] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: stopping tablet replica
I20260620 21:59:17.594511 15165 raft_consensus.cc:2229] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:17.594511 15163 raft_consensus.cc:2229] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:17.594521 15164 raft_consensus.cc:2229] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:17.594552 15165 raft_consensus.cc:2258] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.594554 15163 raft_consensus.cc:2258] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.594585 15164 raft_consensus.cc:2258] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.594921 15164 ts_tablet_manager.cc:1905] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.594928 15165 ts_tablet_manager.cc:1905] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.596740 15165 ts_tablet_manager.cc:1918] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:59:17.596814 15165 log.cc:1196] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/8f98bc7c649847979aca89cdc62470be
I20260620 21:59:17.596887 15164 ts_tablet_manager.cc:1918] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:59:17.596958 15164 log.cc:1196] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/8f98bc7c649847979aca89cdc62470be
I20260620 21:59:17.597056 15165 ts_tablet_manager.cc:1939] T 8f98bc7c649847979aca89cdc62470be P c09b7a8bef154751927bd6cdefe4a107: Deleting consensus metadata
I20260620 21:59:17.597162 15163 ts_tablet_manager.cc:1905] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.597185 15164 ts_tablet_manager.cc:1939] T 8f98bc7c649847979aca89cdc62470be P f36f11e47daa42778564432d5bd92867: Deleting consensus metadata
I20260620 21:59:17.597481 14615 catalog_manager.cc:4841] TS c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809): tablet 8f98bc7c649847979aca89cdc62470be (table test-table [id=5e795a3ee0e8425092a22323a9449f9c]) successfully deleted
I20260620 21:59:17.597582 14612 catalog_manager.cc:4841] TS f36f11e47daa42778564432d5bd92867 (127.6.241.129:43489): tablet 8f98bc7c649847979aca89cdc62470be (table test-table [id=5e795a3ee0e8425092a22323a9449f9c]) successfully deleted
I20260620 21:59:17.598685 15163 ts_tablet_manager.cc:1918] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:59:17.598757 15163 log.cc:1196] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/8f98bc7c649847979aca89cdc62470be
I20260620 21:59:17.599009 15163 ts_tablet_manager.cc:1939] T 8f98bc7c649847979aca89cdc62470be P 9ee31c311603460d9f8c9f9040513102: Deleting consensus metadata
I20260620 21:59:17.599396 14615 catalog_manager.cc:4841] TS 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343): tablet 8f98bc7c649847979aca89cdc62470be (table test-table [id=5e795a3ee0e8425092a22323a9449f9c]) successfully deleted
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:59:17.625295 14692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:17.619055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.6.241.190:33021)
0620 21:59:17.619123 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.619126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.619263 (+ 137us) server_negotiation.cc:407] Connection header received
0620 21:59:17.619353 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.619354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.619365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.619420 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:17.620235 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.620820 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.621989 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.622188 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.622292 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.625215 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44382: BlockingRecv error: recv got EOF from 127.0.0.1:44382 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 21:59:17 dist-test-slave-gl2g krb5kdc[14583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM
I20260620 21:59:17.633502 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.626227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.6.241.190:33021)
0620 21:59:17.626285 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.626287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.626297 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:17.626376 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.626377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.626386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.626444 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:17.626990 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.627480 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.628194 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.628401 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.628477 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.630970 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:17.630972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:17.630972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:17.630982 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:17.632363 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.632717 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.632718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.632718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.632778 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.632974 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.632975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.632975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.633094 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:17.633197 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.633302 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.633392 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 21:59:17.635262 14627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:17.635479 14627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:17.636960 14905 tablet_service.cc:1467] Processing CreateTablet for tablet 23198a7b573d4c108816c9139c234a9c (DEFAULT_TABLE table=test-table [id=ccf1e08796624e7db192436ef5605b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.637066 15045 tablet_service.cc:1467] Processing CreateTablet for tablet 23198a7b573d4c108816c9139c234a9c (DEFAULT_TABLE table=test-table [id=ccf1e08796624e7db192436ef5605b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.637055 14765 tablet_service.cc:1467] Processing CreateTablet for tablet 23198a7b573d4c108816c9139c234a9c (DEFAULT_TABLE table=test-table [id=ccf1e08796624e7db192436ef5605b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.637120 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23198a7b573d4c108816c9139c234a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.637167 15045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23198a7b573d4c108816c9139c234a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.637225 14765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23198a7b573d4c108816c9139c234a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.638468 15140 tablet_bootstrap.cc:492] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: Bootstrap starting.
I20260620 21:59:17.638525 15139 tablet_bootstrap.cc:492] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: Bootstrap starting.
I20260620 21:59:17.638942 15140 tablet_bootstrap.cc:654] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.639065 15139 tablet_bootstrap.cc:654] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.639319 15141 tablet_bootstrap.cc:492] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: Bootstrap starting.
I20260620 21:59:17.639644 15139 tablet_bootstrap.cc:492] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: No bootstrap required, opened a new log
I20260620 21:59:17.639695 15139 ts_tablet_manager.cc:1397] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:17.639720 15141 tablet_bootstrap.cc:654] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.639798 15139 raft_consensus.cc:348] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.639878 15139 raft_consensus.cc:374] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.639897 15139 raft_consensus.cc:729] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36f11e47daa42778564432d5bd92867, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.639936 15139 consensus_queue.cc:260] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.640017 15139 ts_tablet_manager.cc:1428] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.640157 15140 tablet_bootstrap.cc:492] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: No bootstrap required, opened a new log
I20260620 21:59:17.640218 15140 ts_tablet_manager.cc:1397] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:59:17.640380 15140 raft_consensus.cc:348] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.640452 15140 raft_consensus.cc:374] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.640486 15140 raft_consensus.cc:729] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee31c311603460d9f8c9f9040513102, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.640539 15140 consensus_queue.cc:260] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.640625 15140 ts_tablet_manager.cc:1428] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.641328 15141 tablet_bootstrap.cc:492] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: No bootstrap required, opened a new log
I20260620 21:59:17.641382 15141 ts_tablet_manager.cc:1397] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:17.641546 15141 raft_consensus.cc:348] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.641638 15141 raft_consensus.cc:374] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.641672 15141 raft_consensus.cc:729] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09b7a8bef154751927bd6cdefe4a107, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.641723 15141 consensus_queue.cc:260] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.641819 15141 ts_tablet_manager.cc:1428] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:17.772446 15150 raft_consensus.cc:482] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:17.772560 15150 raft_consensus.cc:504] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.772765 15150 leader_election.cc:288] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809), 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343)
I20260620 21:59:17.773031 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23198a7b573d4c108816c9139c234a9c" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee31c311603460d9f8c9f9040513102" is_pre_election: true
I20260620 21:59:17.773022 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23198a7b573d4c108816c9139c234a9c" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09b7a8bef154751927bd6cdefe4a107" is_pre_election: true
I20260620 21:59:17.773138 15065 raft_consensus.cc:2454] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 0.
I20260620 21:59:17.773138 14925 raft_consensus.cc:2454] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 0.
I20260620 21:59:17.773314 14719 leader_election.cc:302] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ee31c311603460d9f8c9f9040513102, f36f11e47daa42778564432d5bd92867; no voters:
I20260620 21:59:17.773424 15150 raft_consensus.cc:2781] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:17.773468 15150 raft_consensus.cc:482] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:17.773492 15150 raft_consensus.cc:3037] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.774257 15150 raft_consensus.cc:504] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.774398 15150 leader_election.cc:288] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 election: Requested vote from peers c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809), 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343)
I20260620 21:59:17.774580 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23198a7b573d4c108816c9139c234a9c" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee31c311603460d9f8c9f9040513102"
I20260620 21:59:17.774650 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23198a7b573d4c108816c9139c234a9c" candidate_uuid: "f36f11e47daa42778564432d5bd92867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09b7a8bef154751927bd6cdefe4a107"
I20260620 21:59:17.774695 15065 raft_consensus.cc:3037] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.774734 14925 raft_consensus.cc:3037] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.775269 15065 raft_consensus.cc:2454] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 1.
I20260620 21:59:17.775429 14925 raft_consensus.cc:2454] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36f11e47daa42778564432d5bd92867 in term 1.
I20260620 21:59:17.775497 14719 leader_election.cc:302] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ee31c311603460d9f8c9f9040513102, f36f11e47daa42778564432d5bd92867; no voters:
I20260620 21:59:17.775604 15150 raft_consensus.cc:2781] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:17.775686 15150 raft_consensus.cc:686] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 1 LEADER]: Becoming Leader. State: Replica: f36f11e47daa42778564432d5bd92867, State: Running, Role: LEADER
I20260620 21:59:17.775749 15150 consensus_queue.cc:237] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.776362 14628 catalog_manager.cc:5495] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 reported cstate change: term changed from 0 to 1, leader changed from <none> to f36f11e47daa42778564432d5bd92867 (127.6.241.129). New cstate: current_term: 1 leader_uuid: "f36f11e47daa42778564432d5bd92867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } health_report { overall_health: UNKNOWN } } }
I20260620 21:59:17.790091 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.788061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.6.241.129:43489)
0620 21:59:17.788124 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.788126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.788135 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:17.788163 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.788163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.788176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.788227 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:17.788350 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.788824 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.789564 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.789741 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.789789 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.789903 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.789931 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.789967 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.790010 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.790030 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 21:59:17.790989 14925 raft_consensus.cc:1261] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Refusing update from remote peer f36f11e47daa42778564432d5bd92867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.790984 15065 raft_consensus.cc:1261] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Refusing update from remote peer f36f11e47daa42778564432d5bd92867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.791232 15146 consensus_queue.cc:1035] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.791379 15150 consensus_queue.cc:1035] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.793674 14628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:17.793769 14628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:17.794389 14628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: Sending DeleteTablet for 3 replicas of tablet 23198a7b573d4c108816c9139c234a9c
I20260620 21:59:17.794684 14905 tablet_service.cc:1514] Processing DeleteTablet for tablet 23198a7b573d4c108816c9139c234a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:45982
I20260620 21:59:17.794814 14765 tablet_service.cc:1514] Processing DeleteTablet for tablet 23198a7b573d4c108816c9139c234a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:53208
I20260620 21:59:17.794821 15165 tablet_replica.cc:330] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: stopping tablet replica
I20260620 21:59:17.794893 15165 raft_consensus.cc:2229] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:17.794905 15164 tablet_replica.cc:330] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: stopping tablet replica
I20260620 21:59:17.794960 15165 raft_consensus.cc:2258] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.794960 15164 raft_consensus.cc:2229] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:17.794956 15045 tablet_service.cc:1514] Processing DeleteTablet for tablet 23198a7b573d4c108816c9139c234a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:35142
I20260620 21:59:17.795027 15164 raft_consensus.cc:2258] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.795045 15163 tablet_replica.cc:330] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: stopping tablet replica
I20260620 21:59:17.795101 15163 raft_consensus.cc:2229] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:17.795152 15163 raft_consensus.cc:2258] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.795349 14628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:59:17.795424 15163 ts_tablet_manager.cc:1905] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:59:17.795497 14628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:17.795495 15165 ts_tablet_manager.cc:1905] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.796657 14765 tablet_service.cc:1467] Processing CreateTablet for tablet eb01cc9213e448ef9e42255726473847 (DEFAULT_TABLE table=test-table [id=b930677c23cf42b5a0aab0c04a5d2c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.796772 14765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb01cc9213e448ef9e42255726473847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.797152 15163 ts_tablet_manager.cc:1918] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:17.797199 15164 ts_tablet_manager.cc:1905] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.797219 15163 log.cc:1196] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/23198a7b573d4c108816c9139c234a9c
I20260620 21:59:17.797386 15163 ts_tablet_manager.cc:1939] T 23198a7b573d4c108816c9139c234a9c P 9ee31c311603460d9f8c9f9040513102: Deleting consensus metadata
I20260620 21:59:17.798240 15139 tablet_bootstrap.cc:492] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867: Bootstrap starting.
I20260620 21:59:17.798246 14905 tablet_service.cc:1467] Processing CreateTablet for tablet eb01cc9213e448ef9e42255726473847 (DEFAULT_TABLE table=test-table [id=b930677c23cf42b5a0aab0c04a5d2c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.798513 15164 ts_tablet_manager.cc:1918] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:17.798558 15139 tablet_bootstrap.cc:654] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.798561 15164 log.cc:1196] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/23198a7b573d4c108816c9139c234a9c
I20260620 21:59:17.798826 15045 tablet_service.cc:1467] Processing CreateTablet for tablet eb01cc9213e448ef9e42255726473847 (DEFAULT_TABLE table=test-table [id=b930677c23cf42b5a0aab0c04a5d2c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:17.798870 15164 ts_tablet_manager.cc:1939] T 23198a7b573d4c108816c9139c234a9c P f36f11e47daa42778564432d5bd92867: Deleting consensus metadata
I20260620 21:59:17.798914 15165 ts_tablet_manager.cc:1918] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:17.798935 15045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb01cc9213e448ef9e42255726473847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.798971 15165 log.cc:1196] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/23198a7b573d4c108816c9139c234a9c
I20260620 21:59:17.799144 14612 catalog_manager.cc:4841] TS f36f11e47daa42778564432d5bd92867 (127.6.241.129:43489): tablet 23198a7b573d4c108816c9139c234a9c (table test-table [id=ccf1e08796624e7db192436ef5605b0f]) successfully deleted
I20260620 21:59:17.799182 15165 ts_tablet_manager.cc:1939] T 23198a7b573d4c108816c9139c234a9c P c09b7a8bef154751927bd6cdefe4a107: Deleting consensus metadata
I20260620 21:59:17.799309 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb01cc9213e448ef9e42255726473847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.800146 14615 catalog_manager.cc:4841] TS c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809): tablet 23198a7b573d4c108816c9139c234a9c (table test-table [id=ccf1e08796624e7db192436ef5605b0f]) successfully deleted
I20260620 21:59:17.800140 15140 tablet_bootstrap.cc:492] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: Bootstrap starting.
I20260620 21:59:17.800220 14615 catalog_manager.cc:4841] TS 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343): tablet 23198a7b573d4c108816c9139c234a9c (table test-table [id=ccf1e08796624e7db192436ef5605b0f]) successfully deleted
I20260620 21:59:17.800472 15140 tablet_bootstrap.cc:654] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.800544 15141 tablet_bootstrap.cc:492] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: Bootstrap starting.
I20260620 21:59:17.800623 15139 tablet_bootstrap.cc:492] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867: No bootstrap required, opened a new log
I20260620 21:59:17.800679 15139 ts_tablet_manager.cc:1397] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:17.800786 15139 raft_consensus.cc:348] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.800866 15139 raft_consensus.cc:374] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.800891 15139 raft_consensus.cc:729] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36f11e47daa42778564432d5bd92867, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.800938 15139 consensus_queue.cc:260] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.801029 15139 ts_tablet_manager.cc:1428] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.801153 15141 tablet_bootstrap.cc:654] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.801498 15140 tablet_bootstrap.cc:492] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: No bootstrap required, opened a new log
I20260620 21:59:17.801558 15140 ts_tablet_manager.cc:1397] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:17.801669 15140 raft_consensus.cc:348] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.801751 15140 raft_consensus.cc:374] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.801774 15140 raft_consensus.cc:729] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee31c311603460d9f8c9f9040513102, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.801805 15141 tablet_bootstrap.cc:492] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: No bootstrap required, opened a new log
I20260620 21:59:17.801820 15140 consensus_queue.cc:260] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.801853 15141 ts_tablet_manager.cc:1397] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:17.801919 15140 ts_tablet_manager.cc:1428] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.801993 15141 raft_consensus.cc:348] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.802073 15141 raft_consensus.cc:374] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.802098 15141 raft_consensus.cc:729] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09b7a8bef154751927bd6cdefe4a107, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.802141 15141 consensus_queue.cc:260] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.802232 15141 ts_tablet_manager.cc:1428] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.914819 15145 raft_consensus.cc:482] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:17.914958 15145 raft_consensus.cc:504] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.915383 15145 leader_election.cc:288] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f36f11e47daa42778564432d5bd92867 (127.6.241.129:43489), 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343)
I20260620 21:59:17.919402 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.915614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:46477 (local address 127.6.241.131:38343)
0620 21:59:17.915660 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.915662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.915732 (+ 70us) server_negotiation.cc:407] Connection header received
0620 21:59:17.915791 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.915791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.915802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.915872 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:17.916037 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.916804 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.918457 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.919046 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.919119 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.919128 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.919295 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.919329 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:59:17.919402 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.915614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:38439 (local address 127.6.241.129:43489)
0620 21:59:17.915660 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.915662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.915735 (+ 73us) server_negotiation.cc:407] Connection header received
0620 21:59:17.915792 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.915793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.915802 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.915872 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:17.916038 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.916805 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.918458 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.919039 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.919119 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.919128 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.919294 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.919336 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:59:17.919428 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.915518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:38343 (local address 127.6.241.130:46477)
0620 21:59:17.915649 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.915661 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.915731 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.915906 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.915907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.915908 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:17.915964 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.915966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.916947 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.916948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.918364 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.918367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.919127 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.919128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.919198 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.919231 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.919282 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.919318 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":78,"thread_start_us":67,"threads_started":1}
I20260620 21:59:17.919437 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.915518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:43489 (local address 127.6.241.130:38439)
0620 21:59:17.915649 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.915661 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.915735 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.915889 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.915890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.915893 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:17.915969 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.915971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.916947 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.916948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.918364 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.918367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.919128 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.919129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.919198 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.919205 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.919266 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.919318 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":89,"thread_start_us":66,"threads_started":1}
I20260620 21:59:17.919740 14785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01cc9213e448ef9e42255726473847" candidate_uuid: "c09b7a8bef154751927bd6cdefe4a107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36f11e47daa42778564432d5bd92867" is_pre_election: true
I20260620 21:59:17.919747 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01cc9213e448ef9e42255726473847" candidate_uuid: "c09b7a8bef154751927bd6cdefe4a107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee31c311603460d9f8c9f9040513102" is_pre_election: true
I20260620 21:59:17.919842 15065 raft_consensus.cc:2454] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c09b7a8bef154751927bd6cdefe4a107 in term 0.
I20260620 21:59:17.919857 14785 raft_consensus.cc:2454] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c09b7a8bef154751927bd6cdefe4a107 in term 0.
I20260620 21:59:17.920013 14859 leader_election.cc:302] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ee31c311603460d9f8c9f9040513102, c09b7a8bef154751927bd6cdefe4a107; no voters:
I20260620 21:59:17.920150 15145 raft_consensus.cc:2781] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:17.920224 15145 raft_consensus.cc:482] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:17.920260 15145 raft_consensus.cc:3037] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.920975 15145 raft_consensus.cc:504] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.921131 15145 leader_election.cc:288] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [CANDIDATE]: Term 1 election: Requested vote from peers f36f11e47daa42778564432d5bd92867 (127.6.241.129:43489), 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343)
I20260620 21:59:17.921288 14785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01cc9213e448ef9e42255726473847" candidate_uuid: "c09b7a8bef154751927bd6cdefe4a107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36f11e47daa42778564432d5bd92867"
I20260620 21:59:17.921322 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01cc9213e448ef9e42255726473847" candidate_uuid: "c09b7a8bef154751927bd6cdefe4a107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee31c311603460d9f8c9f9040513102"
I20260620 21:59:17.921384 14785 raft_consensus.cc:3037] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.921408 15065 raft_consensus.cc:3037] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.922003 15065 raft_consensus.cc:2454] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c09b7a8bef154751927bd6cdefe4a107 in term 1.
I20260620 21:59:17.922020 14785 raft_consensus.cc:2454] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c09b7a8bef154751927bd6cdefe4a107 in term 1.
I20260620 21:59:17.922186 14856 leader_election.cc:302] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c09b7a8bef154751927bd6cdefe4a107, f36f11e47daa42778564432d5bd92867; no voters:
I20260620 21:59:17.922279 15145 raft_consensus.cc:2781] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:17.922394 15145 raft_consensus.cc:686] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 1 LEADER]: Becoming Leader. State: Replica: c09b7a8bef154751927bd6cdefe4a107, State: Running, Role: LEADER
I20260620 21:59:17.922463 15145 consensus_queue.cc:237] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } }
I20260620 21:59:17.923177 14628 catalog_manager.cc:5495] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 reported cstate change: term changed from 0 to 1, leader changed from <none> to c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130). New cstate: current_term: 1 leader_uuid: "c09b7a8bef154751927bd6cdefe4a107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c09b7a8bef154751927bd6cdefe4a107" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 } health_report { overall_health: UNKNOWN } } }
I20260620 21:59:17.948424 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.946231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.6.241.130:34809)
0620 21:59:17.946275 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.946277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.946284 (+ 7us) server_negotiation.cc:407] Connection header received
0620 21:59:17.946350 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.946351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.946372 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.946429 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:17.946516 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.947054 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.947735 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.947928 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.947979 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.948104 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.948210 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.948264 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.948298 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.948318 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 21:59:17.953040 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.950833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:35097 (local address 127.6.241.131:38343)
0620 21:59:17.950872 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.950873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.950973 (+ 100us) server_negotiation.cc:407] Connection header received
0620 21:59:17.951022 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.951022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.951031 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.951074 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:17.951160 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.951600 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.952275 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.952444 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.952491 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.952862 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.952886 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.952922 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.952971 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.952985 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 21:59:17.953070 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.950756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:38343 (local address 127.6.241.130:35097)
0620 21:59:17.950928 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.950936 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.950983 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.951074 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.951074 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.951076 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:59:17.951127 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.951128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.951653 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.951654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.952208 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.952210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.952724 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.952725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.952768 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.952779 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.952934 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.952936 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.952969 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.953005 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260620 21:59:17.955343 15065 raft_consensus.cc:1261] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Refusing update from remote peer c09b7a8bef154751927bd6cdefe4a107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.955319 14785 raft_consensus.cc:1261] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Refusing update from remote peer c09b7a8bef154751927bd6cdefe4a107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:17.955601 15220 consensus_queue.cc:1035] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f36f11e47daa42778564432d5bd92867" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 43489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.955727 15145 consensus_queue.cc:1035] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee31c311603460d9f8c9f9040513102" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:17.962814 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.960410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:41865 (local address 127.6.241.130:34809)
0620 21:59:17.960451 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.960452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.960461 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:17.960530 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.960531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.960542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.960586 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:17.960736 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.961252 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.961993 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.962221 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.962288 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.962524 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.962560 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.962641 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.962706 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.962733 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 21:59:17.962845 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.960376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:34809 (local address 127.6.241.131:41865)
0620 21:59:17.960419 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.960432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:17.960471 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:17.960623 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:17.960624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:17.960626 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:59:17.960676 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.960678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.961332 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.961333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.961906 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:17.961907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.962350 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.962351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:17.962421 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.962429 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:59:17.962643 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:59:17.962645 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:59:17.962689 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 21:59:17.962740 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 21:59:17.969561 14626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:17.969722 14626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:17.970432 14626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23722ee6c5124c6e9a6e8e3f4cc6a077: Sending DeleteTablet for 3 replicas of tablet eb01cc9213e448ef9e42255726473847
I20260620 21:59:17.970693 14765 tablet_service.cc:1514] Processing DeleteTablet for tablet eb01cc9213e448ef9e42255726473847 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:53208
I20260620 21:59:17.970742 15045 tablet_service.cc:1514] Processing DeleteTablet for tablet eb01cc9213e448ef9e42255726473847 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:35142
I20260620 21:59:17.970800 15164 tablet_replica.cc:330] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867: stopping tablet replica
I20260620 21:59:17.970852 15163 tablet_replica.cc:330] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: stopping tablet replica
I20260620 21:59:17.970857 15164 raft_consensus.cc:2229] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:17.970894 15164 raft_consensus.cc:2258] T eb01cc9213e448ef9e42255726473847 P f36f11e47daa42778564432d5bd92867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.970904 15163 raft_consensus.cc:2229] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:17.970932 15163 raft_consensus.cc:2258] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.971251 15163 ts_tablet_manager.cc:1905] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.971266 14905 tablet_service.cc:1514] Processing DeleteTablet for tablet eb01cc9213e448ef9e42255726473847 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:17 UTC) from {username='oryx', principal='oryx/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:45982
I20260620 21:59:17.971354 15165 tablet_replica.cc:330] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: stopping tablet replica
I20260620 21:59:17.971396 15165 raft_consensus.cc:2229] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:17.971469 15165 raft_consensus.cc:2258] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:17.971485 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 14700
I20260620 21:59:17.972724 15165 ts_tablet_manager.cc:1905] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:17.972986 15163 ts_tablet_manager.cc:1918] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:59:17.973060 15163 log.cc:1196] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/eb01cc9213e448ef9e42255726473847
I20260620 21:59:17.973272 15163 ts_tablet_manager.cc:1939] T eb01cc9213e448ef9e42255726473847 P 9ee31c311603460d9f8c9f9040513102: Deleting consensus metadata
I20260620 21:59:17.973598 14615 catalog_manager.cc:4841] TS 9ee31c311603460d9f8c9f9040513102 (127.6.241.131:38343): tablet eb01cc9213e448ef9e42255726473847 (table test-table [id=b930677c23cf42b5a0aab0c04a5d2c0b]) successfully deleted
I20260620 21:59:17.974162 15165 ts_tablet_manager.cc:1918] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:59:17.974227 15165 log.cc:1196] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/eb01cc9213e448ef9e42255726473847
I20260620 21:59:17.974440 15165 ts_tablet_manager.cc:1939] T eb01cc9213e448ef9e42255726473847 P c09b7a8bef154751927bd6cdefe4a107: Deleting consensus metadata
I20260620 21:59:17.975613 14615 catalog_manager.cc:4841] TS c09b7a8bef154751927bd6cdefe4a107 (127.6.241.130:34809): tablet eb01cc9213e448ef9e42255726473847 (table test-table [id=b930677c23cf42b5a0aab0c04a5d2c0b]) successfully deleted
W20260620 21:59:17.979970 14612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:43489 (error 108)
I20260620 21:59:17.980211 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 14840
I20260620 21:59:17.980942 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.980367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:43489 (local address 127.0.0.1:53248)
0620 21:59:17.980844 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:17.980891 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:43489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":37,"threads_started":1}
W20260620 21:59:17.981077 14612 catalog_manager.cc:4584] TS f36f11e47daa42778564432d5bd92867 (127.6.241.129:43489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb01cc9213e448ef9e42255726473847: Network error: Client connection negotiation failed: client connection to 127.6.241.129:43489: connect: Connection refused (error 111)
I20260620 21:59:17.986670 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 14980
I20260620 21:59:17.993024 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 14596
2026-06-20T21:59:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2817 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:18 dist-test-slave-gl2g krb5kdc[15233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:18 dist-test-slave-gl2g krb5kdc[15233](info): set up 2 sockets
Jun 20 21:59:18 dist-test-slave-gl2g krb5kdc[15233](info): commencing operation
krb5kdc: starting...
W20260620 21:59:19.047605 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:59:19 dist-test-slave-gl2g krb5kdc[15233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:59:19.171317 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:46833
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:46833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:19.252413 15246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:19.252642 15246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:19.252665 15246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:19.253772 15246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:19.253804 15246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:19.253818 15246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:19.253831 15246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:19.253844 15246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:19.255327 15246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:46833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:19.255546 15246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:19.255810 15246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:19.260725 15251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:19.260779 15246 server_base.cc:1034] running on GCE node
W20260620 21:59:19.260732 15254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:19.260744 15252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:19.261186 15246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:19.261502 15246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:19.262663 15246 hybrid_clock.cc:648] HybridClock initialized: now 1781992759262640 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T21:59:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1274 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:19 dist-test-slave-gl2g krb5kdc[15259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:19 dist-test-slave-gl2g krb5kdc[15259](info): set up 2 sockets
Jun 20 21:59:19 dist-test-slave-gl2g krb5kdc[15259](info): commencing operation
krb5kdc: starting...
W20260620 21:59:20.311321 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:59:20 dist-test-slave-gl2g krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:59:20.429121 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42295
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:42295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:20.510916 15272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:20.511132 15272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:20.511157 15272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:20.512333 15272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:20.512384 15272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:20.512399 15272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:20.512411 15272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:20.512424 15272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:20.513975 15272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:42295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:20.514225 15272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:20.514454 15272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:20.519291 15280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:20.519341 15278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:20.519328 15277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:20.519816 15272 server_base.cc:1034] running on GCE node
I20260620 21:59:20.520102 15272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:20.520421 15272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:20.521595 15272 hybrid_clock.cc:648] HybridClock initialized: now 1781992760521580 us; error 44 us; skew 500 ppm
Jun 20 21:59:20 dist-test-slave-gl2g krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:20.524206 15272 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:59:20.524667 15272 webserver.cc:460] Webserver started at http://127.6.241.190:32891/ using document root <none> and password file <none>
I20260620 21:59:20.524894 15272 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:20.524964 15272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:20.525117 15272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:20.526189 15272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "f11a6969e6b643568c32070c46958878"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "a16050491c88904858e0ed224b88b4d0"
server_key_iv: "ebfed193ba3d62dff18a17aef37c0b69"
server_key_version: "encryptionkey@0"
I20260620 21:59:20.526559 15272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "f11a6969e6b643568c32070c46958878"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "a16050491c88904858e0ed224b88b4d0"
server_key_iv: "ebfed193ba3d62dff18a17aef37c0b69"
server_key_version: "encryptionkey@0"
I20260620 21:59:20.527930 15272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:20.528817 15287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:20.529031 15272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:20.529129 15272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f11a6969e6b643568c32070c46958878"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "a16050491c88904858e0ed224b88b4d0"
server_key_iv: "ebfed193ba3d62dff18a17aef37c0b69"
server_key_version: "encryptionkey@0"
I20260620 21:59:20.529220 15272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:20.562503 15272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:20.565096 15272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:20.565266 15272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:20.569624 15272 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:42295
I20260620 21:59:20.569731 15339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:42295 every 8 connection(s)
I20260620 21:59:20.570093 15272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:20.571276 15340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:20.574548 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 15272
I20260620 21:59:20.574719 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:20.574920 7110 external_mini_cluster.cc:1428] Setting key 8b4a7a6336a2ba6272cac70861a29efa
I20260620 21:59:20.575047 15340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: Bootstrap starting.
I20260620 21:59:20.575812 15340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:20.576152 15340 log.cc:824] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:20.576916 15340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: No bootstrap required, opened a new log
I20260620 21:59:20.579303 15340 raft_consensus.cc:348] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } }
I20260620 21:59:20.579484 15340 raft_consensus.cc:374] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:20.579530 15340 raft_consensus.cc:729] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11a6969e6b643568c32070c46958878, State: Initialized, Role: FOLLOWER
I20260620 21:59:20.579659 15340 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } }
I20260620 21:59:20.579757 15340 raft_consensus.cc:388] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:20.579794 15340 raft_consensus.cc:482] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:20.579836 15340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:20.580585 15340 raft_consensus.cc:504] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } }
I20260620 21:59:20.580698 15340 leader_election.cc:302] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f11a6969e6b643568c32070c46958878; no voters:
I20260620 21:59:20.580878 15340 leader_election.cc:288] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:20.580924 15345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:20.581084 15345 raft_consensus.cc:686] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 1 LEADER]: Becoming Leader. State: Replica: f11a6969e6b643568c32070c46958878, State: Running, Role: LEADER
I20260620 21:59:20.581152 15340 sys_catalog.cc:564] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:20.581182 15345 consensus_queue.cc:237] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } }
I20260620 21:59:20.581502 15346 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f11a6969e6b643568c32070c46958878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } } }
I20260620 21:59:20.581657 15346 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:20.581965 15353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:20.582515 15353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 21:59:20 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:20.584146 15347 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f11a6969e6b643568c32070c46958878. Latest consensus state: current_term: 1 leader_uuid: "f11a6969e6b643568c32070c46958878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } } }
I20260620 21:59:20.584221 15347 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:20.584976 15362 mvcc.cc:205] Tried to move back new op lower bound from 7299042347350437888 to 7299042347340902400. Current Snapshot: MvccSnapshot[applied={T|T < 7299042347350437888}]
I20260620 21:59:20.587038 15353 catalog_manager.cc:1347] Generated new cluster ID: d467c77b272747128e1557321cd57990
I20260620 21:59:20.587145 15353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:20.588060 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.575499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.6.241.190:42295)
0620 21:59:20.575692 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:20.575701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:20.575739 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:59:20.576317 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:20.576327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:20.576419 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:20.576582 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:20.577382 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.578899 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.580071 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.580705 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.580817 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.583952 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:20.583955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:20.583955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:20.583966 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:20.586018 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:20.586427 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:20.586431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:20.586431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:20.586527 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:20.586770 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:20.586771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:20.586771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:20.587080 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:20.587244 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:20.587716 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 21:59:20.587849 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260620 21:59:20.610647 15353 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:20.611287 15353 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:20.618818 15353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: Generated new TSK 0
I20260620 21:59:20.619050 15353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:59:20.649690 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42295
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:20.732275 15366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:20.732488 15366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:20.732512 15366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:20.733696 15366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:20.733737 15366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:20.733764 15366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:20.735256 15366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:20.735446 15366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:20.735620 15366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:20.740741 15366 server_base.cc:1034] running on GCE node
W20260620 21:59:20.740871 15371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:20.740975 15372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:20.740939 15374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:20.741195 15366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:20.741462 15366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:20.742651 15366 hybrid_clock.cc:648] HybridClock initialized: now 1781992760742636 us; error 45 us; skew 500 ppm
Jun 20 21:59:20 dist-test-slave-gl2g krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:20.745230 15366 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:59:20.745661 15366 webserver.cc:460] Webserver started at http://127.6.241.129:45755/ using document root <none> and password file <none>
I20260620 21:59:20.745894 15366 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:20.745965 15366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:20.746117 15366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:20.747280 15366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "1cfc45021dda4c7ca4ae8bdc00057443"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "3bda0e758372fc56cf35b78e6775c3d0"
server_key_iv: "8c1dcf4db79a787ff9c84f0faaa804c9"
server_key_version: "encryptionkey@0"
I20260620 21:59:20.747646 15366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "1cfc45021dda4c7ca4ae8bdc00057443"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "3bda0e758372fc56cf35b78e6775c3d0"
server_key_iv: "8c1dcf4db79a787ff9c84f0faaa804c9"
server_key_version: "encryptionkey@0"
I20260620 21:59:20.748962 15366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:20.749737 15381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:20.749917 15366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:20.750025 15366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "1cfc45021dda4c7ca4ae8bdc00057443"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "3bda0e758372fc56cf35b78e6775c3d0"
server_key_iv: "8c1dcf4db79a787ff9c84f0faaa804c9"
server_key_version: "encryptionkey@0"
I20260620 21:59:20.750110 15366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:20.768455 15366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:20.770957 15366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:20.771119 15366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:20.771373 15366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:20.771701 15366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:20.771755 15366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:20.771797 15366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:20.771836 15366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:20.777680 15366 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:46751
I20260620 21:59:20.777724 15494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:46751 every 8 connection(s)
I20260620 21:59:20.778059 15366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:59:20 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:20.784765 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 15366
I20260620 21:59:20.784987 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:20.785205 7110 external_mini_cluster.cc:1428] Setting key 11f0245fa958d67ce51f9da44d5fe9fa
I20260620 21:59:20.787618 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.778876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:53265 (local address 127.6.241.190:42295)
0620 21:59:20.778918 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:20.778920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:20.779137 (+ 217us) server_negotiation.cc:407] Connection header received
0620 21:59:20.779625 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:20.779626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:20.779635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:20.779698 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:20.780631 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.781242 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.782363 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.782559 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.782661 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.784792 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:20.784793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:20.784794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:20.784801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:20.786150 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:20.786589 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:20.786590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:20.786591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:20.786676 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:20.786944 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:20.786945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:20.786945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:20.787071 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:20.787149 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:20.787431 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 21:59:20.787526 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 21:59:20.787830 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.778829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:53265)
0620 21:59:20.779004 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:20.779047 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:20.779543 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:20.779754 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:20.779758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:20.780334 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:20.780541 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:20.780546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.781323 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.781323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:20.782278 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:20.782280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.782669 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.782670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:20.782824 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.783342 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:20.783351 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:20.784688 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:20.786252 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:20.786259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:20.786261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:20.786499 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:20.786754 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:20.786755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:20.786755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:20.786875 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:20.787139 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:20.787146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:20.787294 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 21:59:20.787410 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 21:59:20.787574 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260620 21:59:20.788353 15495 heartbeater.cc:347] Connected to a master server at 127.6.241.190:42295
I20260620 21:59:20.788468 15495 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:20.788652 15495 heartbeater.cc:511] Master 127.6.241.190:42295 requested a full tablet report, sending...
I20260620 21:59:20.789248 15304 ts_manager.cc:194] Registered new tserver with Master: 1cfc45021dda4c7ca4ae8bdc00057443 (127.6.241.129:46751)
I20260620 21:59:20.789952 15304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:53265
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:59:20.809271 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42295
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:20.891439 15500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:20.891677 15500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:20.891705 15500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:20.892976 15500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:20.893029 15500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:20.893059 15500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:20.894629 15500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:20.894899 15500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:20.895145 15500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:20.900480 15506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:20.900462 15505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:20.900461 15508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:20.900741 15500 server_base.cc:1034] running on GCE node
I20260620 21:59:20.901001 15500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:20.901284 15500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:20.902462 15500 hybrid_clock.cc:648] HybridClock initialized: now 1781992760902441 us; error 35 us; skew 500 ppm
Jun 20 21:59:20 dist-test-slave-gl2g krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:20.905215 15500 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:59:20.905628 15500 webserver.cc:460] Webserver started at http://127.6.241.130:42103/ using document root <none> and password file <none>
I20260620 21:59:20.905815 15500 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:20.905855 15500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:20.905966 15500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:20.907071 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "7fa7451c1e8a44ae8d2bca81a06bfe83"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "9b3751549a5c34f3963cf5517847ff3b"
server_key_iv: "81ca1e5c980ec747d47403cda06e0fc0"
server_key_version: "encryptionkey@0"
I20260620 21:59:20.907379 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "7fa7451c1e8a44ae8d2bca81a06bfe83"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "9b3751549a5c34f3963cf5517847ff3b"
server_key_iv: "81ca1e5c980ec747d47403cda06e0fc0"
server_key_version: "encryptionkey@0"
I20260620 21:59:20.908686 15500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:20.909459 15515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:20.909647 15500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:20.909757 15500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "7fa7451c1e8a44ae8d2bca81a06bfe83"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "9b3751549a5c34f3963cf5517847ff3b"
server_key_iv: "81ca1e5c980ec747d47403cda06e0fc0"
server_key_version: "encryptionkey@0"
I20260620 21:59:20.909839 15500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:20.925030 15500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:20.927625 15500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:20.927806 15500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:20.928076 15500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:20.928402 15500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:20.928457 15500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:20.928498 15500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:20.928527 15500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:20.934844 15500 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:45353
I20260620 21:59:20.934890 15628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:45353 every 8 connection(s)
I20260620 21:59:20.935245 15500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
Jun 20 21:59:20 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:20.944468 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.935991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:38961 (local address 127.6.241.190:42295)
0620 21:59:20.936042 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:20.936044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:20.936273 (+ 229us) server_negotiation.cc:407] Connection header received
0620 21:59:20.936808 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:20.936809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:20.936819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:20.936922 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:20.937926 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.938426 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.939500 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.939685 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.939744 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.941929 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:20.941931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:20.941931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:20.941939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:20.943234 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:20.943600 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:20.943601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:20.943601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:20.943649 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:20.943866 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:20.943866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:20.943866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:20.943966 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:20.944015 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:20.944338 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 21:59:20.944399 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 21:59:20.944451 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 15500
I20260620 21:59:20.944586 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:20.944780 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.935960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:38961)
0620 21:59:20.936132 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:20.936176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:20.936713 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:20.936971 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:20.936975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:20.937600 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:20.937822 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:20.937827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.938513 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.938514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:20.939416 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:20.939418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.939763 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.939763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:20.939867 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.940375 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:20.940384 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:20.941837 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:20.943307 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:20.943313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:20.943315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:20.943537 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:20.943707 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:20.943707 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:20.943708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:20.943813 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:20.944021 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:20.944025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:20.944147 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 21:59:20.944324 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 21:59:20.944510 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260620 21:59:20.944825 7110 external_mini_cluster.cc:1428] Setting key b11d7b7eb0761ed9bc16df7b526dd511
I20260620 21:59:20.945369 15629 heartbeater.cc:347] Connected to a master server at 127.6.241.190:42295
I20260620 21:59:20.945483 15629 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:20.945678 15629 heartbeater.cc:511] Master 127.6.241.190:42295 requested a full tablet report, sending...
I20260620 21:59:20.946123 15303 ts_manager.cc:194] Registered new tserver with Master: 7fa7451c1e8a44ae8d2bca81a06bfe83 (127.6.241.130:45353)
I20260620 21:59:20.946504 15303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:38961
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:59:20.969929 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42295
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:21.053092 15634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:21.053323 15634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:21.053349 15634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:21.054652 15634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:21.054709 15634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:21.054766 15634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:21.056329 15634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:42295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:21.056598 15634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:21.056850 15634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:21.061951 15642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.061956 15640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.062003 15639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:21.062201 15634 server_base.cc:1034] running on GCE node
I20260620 21:59:21.062522 15634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:21.062821 15634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:21.063987 15634 hybrid_clock.cc:648] HybridClock initialized: now 1781992761063975 us; error 26 us; skew 500 ppm
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:21.066700 15634 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:59:21.067092 15634 webserver.cc:460] Webserver started at http://127.6.241.131:38657/ using document root <none> and password file <none>
I20260620 21:59:21.067287 15634 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:21.067327 15634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:21.067437 15634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:21.068461 15634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "6e3de66c68f04e0389ff58fd4e4216f0"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-gl2g"
server_key: "af55251f97e8a46e3811727dabe847ff"
server_key_iv: "cc4db0ada63a3492be55b6aeb50137af"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.068760 15634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "6e3de66c68f04e0389ff58fd4e4216f0"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-gl2g"
server_key: "af55251f97e8a46e3811727dabe847ff"
server_key_iv: "cc4db0ada63a3492be55b6aeb50137af"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.070015 15634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:21.070751 15649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.070926 15634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:21.071024 15634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "6e3de66c68f04e0389ff58fd4e4216f0"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-gl2g"
server_key: "af55251f97e8a46e3811727dabe847ff"
server_key_iv: "cc4db0ada63a3492be55b6aeb50137af"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.071111 15634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:21.084390 15634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:21.086264 15634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:21.086428 15634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:21.086692 15634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:21.087015 15634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:21.087075 15634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.087116 15634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:21.087144 15634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.092955 15634 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:44977
I20260620 21:59:21.093065 15762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:44977 every 8 connection(s)
I20260620 21:59:21.093355 15634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:21.094761 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 15634
I20260620 21:59:21.094866 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:21.095063 7110 external_mini_cluster.cc:1428] Setting key 857f0f35bdc28e44123b585781c26dd5
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:21.103989 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.094072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:40551 (local address 127.6.241.190:42295)
0620 21:59:21.094136 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.094138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.094387 (+ 249us) server_negotiation.cc:407] Connection header received
0620 21:59:21.094963 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.094964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.094976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.095032 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.096294 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.096898 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.098285 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.098706 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.098784 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.101271 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:21.101272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:21.101273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:21.101280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:21.102861 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.103215 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.103216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.103217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.103264 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.103476 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.103477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.103477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.103579 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:21.103629 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:21.103859 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 21:59:21.103920 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:59:21.104219 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.094047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:40551)
0620 21:59:21.094218 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.094285 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.094841 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.095112 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.095116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.095980 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.096185 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.096190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.096998 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.096999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.098176 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.098180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.098803 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.098804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.098960 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.099610 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.099618 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.101188 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:21.102932 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:21.102936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:21.102938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:21.103152 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:21.103319 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:21.103319 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:21.103320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:21.103424 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:21.103632 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:21.103636 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:21.103753 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 21:59:21.103835 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:59:21.103958 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260620 21:59:21.104756 15763 heartbeater.cc:347] Connected to a master server at 127.6.241.190:42295
I20260620 21:59:21.104857 15763 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:21.105014 15763 heartbeater.cc:511] Master 127.6.241.190:42295 requested a full tablet report, sending...
I20260620 21:59:21.105376 15303 ts_manager.cc:194] Registered new tserver with Master: 6e3de66c68f04e0389ff58fd4e4216f0 (127.6.241.131:44977)
I20260620 21:59:21.105760 15303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:40551
I20260620 21:59:21.106916 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab.
Entry for principal oryx/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab.
WARNING: no policy specified for oryx/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab.
Entry for principal oryx/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab.
WARNING: no policy specified for oryx/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab.
Entry for principal oryx/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab.
I20260620 21:59:21.176342 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 15366
I20260620 21:59:21.181159 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 15500
I20260620 21:59:21.185505 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 15634
I20260620 21:59:21.189810 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 15272
I20260620 21:59:21.194955 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42295
--webserver_interface=127.6.241.190
--webserver_port=32891
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:42295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:21.276733 15772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:21.276962 15772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:21.276985 15772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:21.278182 15772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:21.278227 15772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:21.278241 15772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:21.278286 15772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:21.278317 15772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:21.279834 15772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:42295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:21.280107 15772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:21.280344 15772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:21.285213 15780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.285216 15777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.285205 15778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:21.285427 15772 server_base.cc:1034] running on GCE node
I20260620 21:59:21.285670 15772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:21.285991 15772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:21.287142 15772 hybrid_clock.cc:648] HybridClock initialized: now 1781992761287128 us; error 29 us; skew 500 ppm
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:21.289736 15772 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:59:21.290247 15772 webserver.cc:460] Webserver started at http://127.6.241.190:32891/ using document root <none> and password file <none>
I20260620 21:59:21.290493 15772 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:21.290581 15772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:21.292004 15772 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:21.292672 15787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.292840 15772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 21:59:21.292950 15772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f11a6969e6b643568c32070c46958878"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "a16050491c88904858e0ed224b88b4d0"
server_key_iv: "ebfed193ba3d62dff18a17aef37c0b69"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.293300 15772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:21.320948 15772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:21.323052 15772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:21.323228 15772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:21.327317 15772 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:42295
I20260620 21:59:21.327373 15839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:42295 every 8 connection(s)
I20260620 21:59:21.327785 15772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:21.329267 15840 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:59:21.329873 15840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: Bootstrap starting.
I20260620 21:59:21.329867 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 15772
I20260620 21:59:21.330183 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:46751
--local_ip_for_outbound_sockets=127.6.241.129
--tserver_master_addrs=127.6.241.190:42295
--webserver_port=45755
--webserver_interface=127.6.241.129
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.241.129
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:59:21.338181 15840 log.cc:824] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:21.340029 15840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:59:21.340332 15840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: Bootstrap complete.
I20260620 21:59:21.342582 15840 raft_consensus.cc:348] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } }
I20260620 21:59:21.342932 15840 raft_consensus.cc:729] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11a6969e6b643568c32070c46958878, State: Initialized, Role: FOLLOWER
I20260620 21:59:21.343119 15840 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } }
I20260620 21:59:21.343197 15840 raft_consensus.cc:388] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:21.343251 15840 raft_consensus.cc:482] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:21.343304 15840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:59:21.344348 15840 raft_consensus.cc:504] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } }
I20260620 21:59:21.344483 15840 leader_election.cc:302] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f11a6969e6b643568c32070c46958878; no voters:
I20260620 21:59:21.344691 15840 leader_election.cc:288] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:59:21.344728 15844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:59:21.344843 15844 raft_consensus.cc:686] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [term 2 LEADER]: Becoming Leader. State: Replica: f11a6969e6b643568c32070c46958878, State: Running, Role: LEADER
I20260620 21:59:21.344970 15840 sys_catalog.cc:564] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:21.344962 15844 consensus_queue.cc:237] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } }
I20260620 21:59:21.345242 15844 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f11a6969e6b643568c32070c46958878. Latest consensus state: current_term: 2 leader_uuid: "f11a6969e6b643568c32070c46958878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } } }
I20260620 21:59:21.345309 15844 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:21.345454 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f11a6969e6b643568c32070c46958878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11a6969e6b643568c32070c46958878" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42295 } } }
I20260620 21:59:21.345530 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:21.345916 15848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:21.346421 15848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:21.347045 15848 catalog_manager.cc:1259] Loaded cluster ID: d467c77b272747128e1557321cd57990
I20260620 21:59:21.347096 15848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:21.348312 15848 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:21.348500 15848 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f11a6969e6b643568c32070c46958878: Loaded TSK: 0
I20260620 21:59:21.348718 15848 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:59:21.415243 15842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:21.415458 15842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:21.415484 15842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:21.416728 15842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:21.416772 15842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:21.416800 15842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:21.418258 15842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.129.keytab
--principal=oryx/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:46751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=45755
--tserver_master_addrs=127.6.241.190:42295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:21.418514 15842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:21.418787 15842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:21.424707 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.424746 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.424772 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:21.425019 15842 server_base.cc:1034] running on GCE node
I20260620 21:59:21.425241 15842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:21.425540 15842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:21.426676 15842 hybrid_clock.cc:648] HybridClock initialized: now 1781992761426661 us; error 27 us; skew 500 ppm
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:21.429752 15842 init.cc:377] Logged in from keytab as oryx/127.6.241.129@KRBTEST.COM (short username oryx)
I20260620 21:59:21.430292 15842 webserver.cc:460] Webserver started at http://127.6.241.129:45755/ using document root <none> and password file <none>
I20260620 21:59:21.430560 15842 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:21.430668 15842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:21.432126 15842 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:21.432796 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.432989 15842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:21.433091 15842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "1cfc45021dda4c7ca4ae8bdc00057443"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "3bda0e758372fc56cf35b78e6775c3d0"
server_key_iv: "8c1dcf4db79a787ff9c84f0faaa804c9"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.433452 15842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:21.444612 15842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:21.447239 15842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:21.447400 15842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:21.447643 15842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:21.447950 15842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:21.448001 15842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.448043 15842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:21.448069 15842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.455296 15842 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:46751
I20260620 21:59:21.455380 15989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:46751 every 8 connection(s)
I20260620 21:59:21.455715 15842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:21.464406 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.456499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:39539)
0620 21:59:21.456676 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.456714 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.457283 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.457743 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.457747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.458340 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.458554 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.458560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.459741 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.459742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.460544 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.460546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.461185 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.461186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.461323 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.461898 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.461907 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.464179 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
W20260620 21:59:21.464439 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:21.456787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:39539 (local address 127.6.241.190:42295)
0620 21:59:21.456993 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.457002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.457044 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:59:21.457517 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.457522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.457564 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.457703 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.458731 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.459636 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.460651 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.461079 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.461214 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.464296 (+ 3082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:39539: BlockingRecv error: recv got EOF from 127.6.241.129:39539 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
W20260620 21:59:21.465015 15990 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:42295 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.241.190:42295: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
I20260620 21:59:21.465237 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 15842
I20260620 21:59:21.465613 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:45353
--local_ip_for_outbound_sockets=127.6.241.130
--tserver_master_addrs=127.6.241.190:42295
--webserver_port=42103
--webserver_interface=127.6.241.130
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.241.130
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:21.470460 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.465341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:43871)
0620 21:59:21.465446 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.465460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.465506 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.465671 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.465672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.465859 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.465926 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.465929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.466864 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.466865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.467690 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.467693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.468128 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.468129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.468204 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.468677 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.468682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.470374 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 21:59:21.470522 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:21.465467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:43871 (local address 127.6.241.190:42295)
0620 21:59:21.465510 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.465512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.465523 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:59:21.465570 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.465571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.465583 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.465650 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.466015 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.466779 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.467788 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.468042 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.468131 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.470458 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:43871: BlockingRecv error: recv got EOF from 127.6.241.129:43871 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:21.475010 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.470949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:46995)
0620 21:59:21.471002 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.471014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.471060 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.471204 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.471205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.471332 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.471357 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.471359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.471994 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.471995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.472497 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.472498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.472838 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.472839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.472890 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.473316 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.473321 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.474930 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 21:59:21.475049 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:21.471019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:46995 (local address 127.6.241.190:42295)
0620 21:59:21.471055 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.471057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.471065 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:21.471123 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.471124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.471135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.471201 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.471436 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.471902 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.472576 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.472759 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.472839 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.474990 (+ 2151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:46995: BlockingRecv error: recv got EOF from 127.6.241.129:46995 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 21:59:21.475373 15990 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:59:21.552496 15994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:21.552708 15994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:21.552733 15994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:21.553874 15994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:21.553915 15994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:21.553941 15994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:21.555362 15994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.130.keytab
--principal=oryx/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:45353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=42103
--tserver_master_addrs=127.6.241.190:42295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:21.555569 15994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:21.555753 15994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:21.561008 16002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.560998 15999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:21.561030 15994 server_base.cc:1034] running on GCE node
W20260620 21:59:21.560998 16000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:21.561419 15994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:21.561694 15994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:21.562858 15994 hybrid_clock.cc:648] HybridClock initialized: now 1781992761562847 us; error 30 us; skew 500 ppm
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:21.565343 15994 init.cc:377] Logged in from keytab as oryx/127.6.241.130@KRBTEST.COM (short username oryx)
I20260620 21:59:21.565820 15994 webserver.cc:460] Webserver started at http://127.6.241.130:42103/ using document root <none> and password file <none>
I20260620 21:59:21.566059 15994 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:21.566145 15994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:21.567570 15994 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:21.568154 16009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.568318 15994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:21.568428 15994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "7fa7451c1e8a44ae8d2bca81a06bfe83"
format_stamp: "Formatted at 2026-06-20 21:59:20 on dist-test-slave-gl2g"
server_key: "9b3751549a5c34f3963cf5517847ff3b"
server_key_iv: "81ca1e5c980ec747d47403cda06e0fc0"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.568779 15994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:21.581153 15994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:21.583779 15994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:21.583905 15994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:21.584187 15994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:21.584520 15994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:21.584560 15994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.584583 15994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:21.584623 15994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.590567 15994 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:45353
I20260620 21:59:21.590692 16122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:45353 every 8 connection(s)
I20260620 21:59:21.591073 15994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:21.592327 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 15994
I20260620 21:59:21.592612 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:44977
--local_ip_for_outbound_sockets=127.6.241.131
--tserver_master_addrs=127.6.241.190:42295
--webserver_port=38657
--webserver_interface=127.6.241.131
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.241.131
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:59:21.600170 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:21.591924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:44629 (local address 127.6.241.190:42295)
0620 21:59:21.591984 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.591986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.592242 (+ 256us) server_negotiation.cc:407] Connection header received
0620 21:59:21.592885 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.592886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.592903 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.592980 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.594077 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.594569 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.596219 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.596526 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.596593 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.600110 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:44629: BlockingRecv error: recv got EOF from 127.6.241.130:44629 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 21:59:21.600246 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.591927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:44629)
0620 21:59:21.592100 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.592138 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.592775 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.593038 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.593042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.593623 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.593963 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.593971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.594719 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.594720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.596099 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.596103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.596619 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.596620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.596802 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.597421 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.597433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.600024 (+ 2591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
W20260620 21:59:21.600477 16123 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:42295 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.241.190:42295: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:21.606053 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.600718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:50743)
0620 21:59:21.600889 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.600898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.600954 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.601102 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.601103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.601218 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.601268 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.601270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.602259 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.602260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.602862 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.602865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.603389 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.603390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.603438 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.603974 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.603979 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.605969 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
W20260620 21:59:21.606151 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:21.600797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:50743 (local address 127.6.241.190:42295)
0620 21:59:21.600861 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.600864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.600961 (+ 97us) server_negotiation.cc:407] Connection header received
0620 21:59:21.601001 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.601002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.601015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.601081 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.601338 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.602156 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.602954 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.603305 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.603388 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.606068 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:50743: BlockingRecv error: recv got EOF from 127.6.241.130:50743 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:21.610606 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.606432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:56769)
0620 21:59:21.606479 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.606497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.606558 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.606718 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.606719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.606849 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.606880 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.606882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.607524 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.607525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.608029 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.608031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.608378 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.608379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.608432 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.608881 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.608885 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.610517 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 21:59:21.610693 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:21.606485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:56769 (local address 127.6.241.190:42295)
0620 21:59:21.606546 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.606549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.606558 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:21.606630 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.606631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.606641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.606714 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.606959 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.607430 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.608114 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.608296 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.608379 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.610589 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:56769: BlockingRecv error: recv got EOF from 127.6.241.130:56769 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 21:59:21.610772 16123 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:59:21.678797 16126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:21.679024 16126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:21.679052 16126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:21.680760 16126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:21.680832 16126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:21.680884 16126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:21.683450 16126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/krb5kdc/oryx_127.6.241.131.keytab
--principal=oryx/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:44977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=38657
--tserver_master_addrs=127.6.241.190:42295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:21.683781 16126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:21.684059 16126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:21.690565 16135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.690550 16132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.691040 16133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:21.691218 16126 server_base.cc:1034] running on GCE node
I20260620 21:59:21.691538 16126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:21.691898 16126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:21.693068 16126 hybrid_clock.cc:648] HybridClock initialized: now 1781992761693001 us; error 78 us; skew 500 ppm
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:21.696712 16126 init.cc:377] Logged in from keytab as oryx/127.6.241.131@KRBTEST.COM (short username oryx)
I20260620 21:59:21.697331 16126 webserver.cc:460] Webserver started at http://127.6.241.131:38657/ using document root <none> and password file <none>
I20260620 21:59:21.697638 16126 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:21.697724 16126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:21.699733 16126 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:21.700479 16142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.700662 16126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:21.700747 16126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "6e3de66c68f04e0389ff58fd4e4216f0"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-gl2g"
server_key: "af55251f97e8a46e3811727dabe847ff"
server_key_iv: "cc4db0ada63a3492be55b6aeb50137af"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.701222 16126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:21.724983 16126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:21.727631 16126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:21.727813 16126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:21.728092 16126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:21.728422 16126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:21.728477 16126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.728520 16126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:21.728555 16126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.735862 16126 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:44977
I20260620 21:59:21.735917 16255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:44977 every 8 connection(s)
I20260620 21:59:21.736377 16126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:21.738473 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 16126
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:21.745633 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.737223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:40475)
0620 21:59:21.737386 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.737441 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.738157 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.738552 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.738558 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.739225 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.739435 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.739440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.740152 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.740153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.741380 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.741384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.741898 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.741899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.742052 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.743060 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.743069 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.745471 (+ 2402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260620 21:59:21.745711 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:21.737509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:40475 (local address 127.6.241.190:42295)
0620 21:59:21.737562 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.737564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.737609 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:59:21.738314 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.738315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.738331 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.738403 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.739545 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.740052 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.741485 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.741769 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.741808 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.745619 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:40475: BlockingRecv error: recv got EOF from 127.6.241.131:40475 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 21:59:21.746138 16256 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:42295 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.241.190:42295: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
I20260620 21:59:21.747370 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.739200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.6.241.190:42295)
0620 21:59:21.739291 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.739292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.739300 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:21.739319 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.739320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.739330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.739428 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.740153 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.740829 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.741642 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.741860 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.741941 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.743448 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:21.743453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:21.743454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:21.743467 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:21.745631 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.746066 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.746070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.746070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.746178 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.746458 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.746459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.746459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.746817 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:21.746974 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:21.747161 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 21:59:21.747250 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":52,"threads_started":1}
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:21.751113 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.746394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:50649)
0620 21:59:21.746446 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.746457 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.746500 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.746675 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.746676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.746815 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.746874 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.746876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.747809 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.747810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.748512 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.748515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.748844 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.748845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.748909 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.749343 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.749348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.751065 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 21:59:21.751243 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:21.746447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:50649 (local address 127.6.241.190:42295)
0620 21:59:21.746490 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.746492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.746501 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:21.746563 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.746564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.746573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.746661 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.746991 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.747731 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.748601 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.748779 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.748841 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.751162 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:50649: BlockingRecv error: recv got EOF from 127.6.241.131:50649 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:21 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:21.755611 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.751412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:58199)
0620 21:59:21.751461 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.751471 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.751512 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.751642 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.751643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.751770 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.751797 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.751799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.752490 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.752491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.752984 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.752986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.753353 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.753354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.753407 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.753857 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.753861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.755524 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 21:59:21.755657 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:21.751464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:58199 (local address 127.6.241.190:42295)
0620 21:59:21.751508 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.751510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.751519 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:21.751573 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.751574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.751583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.751638 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.751898 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.752391 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.753067 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.753265 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.753353 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.755595 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:58199: BlockingRecv error: recv got EOF from 127.6.241.131:58199 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 21:59:21.755757 16256 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 21:59:22 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:22 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:22.481091 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.475898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:34451)
0620 21:59:22.476076 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.476088 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.476143 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.476333 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.476334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.476540 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:22.476592 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.476594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.477320 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.477321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.477899 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.477901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.478249 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.478250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.478291 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.478843 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:22.478852 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:22.481014 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
W20260620 21:59:22.481247 16262 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:22.476038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:34451 (local address 127.6.241.190:42295)
0620 21:59:22.476176 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.476178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.476187 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:22.476215 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.476216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.476226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.476312 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:22.476701 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.477229 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.477989 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.478183 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.478236 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.481146 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:34451: BlockingRecv error: recv got EOF from 127.6.241.129:34451 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
Jun 20 21:59:22 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:22 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:22.617262 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.611294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:57941)
0620 21:59:22.611457 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.611472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.611539 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.611759 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.611761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.611943 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:22.611995 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.611997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.612791 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.612792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.613489 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.613494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.613955 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.613956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.614025 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.614806 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:22.614811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:22.617182 (+ 2371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
W20260620 21:59:22.617479 16262 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:22.611440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:57941 (local address 127.6.241.190:42295)
0620 21:59:22.611522 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.611524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.611537 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:59:22.611594 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.611595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.611606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.611670 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:22.612117 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.612682 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.613612 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.613823 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.613916 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.617342 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:57941: BlockingRecv error: recv got EOF from 127.6.241.130:57941 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 21:59:22 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:22 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:22.761389 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.756193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:53383)
0620 21:59:22.756440 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.756452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.756563 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.756740 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.756742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.756955 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:22.757005 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.757007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.757748 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.757750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.758317 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.758320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.758767 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.758768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.758839 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.759390 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:22.759395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:22.761318 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":169,"threads_started":1}
W20260620 21:59:22.761575 16262 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:22.756299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:53383 (local address 127.6.241.190:42295)
0620 21:59:22.756401 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.756404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.756548 (+ 144us) server_negotiation.cc:407] Connection header received
0620 21:59:22.756627 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.756628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.756642 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.756731 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:22.757099 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.757643 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.758416 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.758673 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.758766 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.761450 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:53383: BlockingRecv error: recv got EOF from 127.6.241.131:53383 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 21:59:23 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:23 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:23.486958 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.481730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:57115)
0620 21:59:23.481911 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.481928 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.482005 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.482191 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.482192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.482360 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:23.482419 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.482421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.483144 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.483145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.483701 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.483702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.484169 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.484170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.484211 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.484789 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:23.484794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:23.486841 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
W20260620 21:59:23.486995 16266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:23.481876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:57115 (local address 127.6.241.190:42295)
0620 21:59:23.482054 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.482056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.482066 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:23.482091 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.482092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.482102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.482165 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:23.482495 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.483066 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.483831 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.484090 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.484158 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.486926 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:57115: BlockingRecv error: recv got EOF from 127.6.241.129:57115 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
Jun 20 21:59:23 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:23 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:23.622489 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.617855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:47513)
0620 21:59:23.618010 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.618021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.618096 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.618268 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.618269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.618457 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:23.618515 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.618517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.619239 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.619240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.619762 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.619763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.620093 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.620094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.620132 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.620615 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:23.620619 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:23.622407 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
W20260620 21:59:23.622563 16266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:23.617904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:47513 (local address 127.6.241.190:42295)
0620 21:59:23.617954 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.617957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.618077 (+ 120us) server_negotiation.cc:407] Connection header received
0620 21:59:23.618159 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.618159 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.618174 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.618247 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:23.618641 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.619162 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.619839 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.620025 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.620079 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.622514 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:47513: BlockingRecv error: recv got EOF from 127.6.241.130:47513 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 21:59:23 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:23 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:23.766780 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.761837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:49283)
0620 21:59:23.762022 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.762036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.762101 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.762239 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.762240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.762425 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:23.762472 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.762474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.763197 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.763198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.763739 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.763741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.764126 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.764127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.764189 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.764698 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:23.764708 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:23.766718 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
W20260620 21:59:23.766939 16266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:23.761970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:49283 (local address 127.6.241.190:42295)
0620 21:59:23.762035 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.762038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.762081 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:59:23.762162 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.762163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.762176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.762233 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:23.762557 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.763106 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.763826 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.764043 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.764116 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.766845 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:49283: BlockingRecv error: recv got EOF from 127.6.241.131:49283 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 21:59:24 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:24 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:24.492134 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:24.487463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:58619)
0620 21:59:24.487601 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:24.487618 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:24.487670 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:24.487851 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:24.487851 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:24.488017 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:24.488070 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:24.488071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.488752 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.488753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:24.489267 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:24.489268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.489638 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.489638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:24.489678 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:24.490226 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:24.490231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:24.492033 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
W20260620 21:59:24.492193 16270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:24.487565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:58619 (local address 127.6.241.190:42295)
0620 21:59:24.487707 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:24.487709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:24.487720 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:59:24.487754 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:24.487754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:24.487764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:24.487840 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:24.488139 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.488684 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.489357 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.489564 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.489625 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:24.492119 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:58619: BlockingRecv error: recv got EOF from 127.6.241.129:58619 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
Jun 20 21:59:24 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:24 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:24.627853 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:24.623048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:42117)
0620 21:59:24.623229 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:24.623241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:24.623317 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:24.623476 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:24.623477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:24.623663 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:24.623711 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:24.623713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.624396 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.624397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:24.624950 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:24.624952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.625324 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.625325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:24.625385 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:24.625903 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:24.625908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:24.627793 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
W20260620 21:59:24.628010 16270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:24.623137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:42117 (local address 127.6.241.190:42295)
0620 21:59:24.623213 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:24.623216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:24.623309 (+ 93us) server_negotiation.cc:407] Connection header received
0620 21:59:24.623380 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:24.623382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:24.623401 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:24.623469 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:24.623794 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.624307 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.625046 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.625236 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.625324 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:24.627940 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:42117: BlockingRecv error: recv got EOF from 127.6.241.130:42117 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 21:59:24 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:24 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:24.772753 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:24.767437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:35785)
0620 21:59:24.767753 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:24.767767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:24.767876 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:24.768082 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:24.768083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:24.768272 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:24.768343 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:24.768345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.769036 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.769037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:24.769570 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:24.769571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.769980 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.769981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:24.770036 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:24.770584 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:24.770636 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:24.772661 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":203,"threads_started":1}
W20260620 21:59:24.772900 16270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:24.767650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:35785 (local address 127.6.241.190:42295)
0620 21:59:24.767743 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:24.767746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:24.767853 (+ 107us) server_negotiation.cc:407] Connection header received
0620 21:59:24.767969 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:24.767970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:24.767990 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:24.768066 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:24.768433 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.768943 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.769661 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.769905 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.769982 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:24.772766 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:35785: BlockingRecv error: recv got EOF from 127.6.241.131:35785 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 21:59:25 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:25 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:25.501325 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.492717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:55893)
0620 21:59:25.492898 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.492909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.492966 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.493104 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.493105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.493264 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:25.493314 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.493315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.494025 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.494025 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.494567 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.494569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.495003 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.495004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.495056 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.499095 (+ 4039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:25.499102 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:25.501262 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
W20260620 21:59:25.501443 16274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:25.492833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:55893 (local address 127.6.241.190:42295)
0620 21:59:25.492960 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.492962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.492971 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:25.493015 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.493016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.493027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.493087 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:25.493429 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.493951 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.494712 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.494921 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.494980 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.501381 (+ 6401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:55893: BlockingRecv error: recv got EOF from 127.6.241.129:55893 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
Jun 20 21:59:25 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:25 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:25.633320 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.628414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:39759)
0620 21:59:25.628553 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.628566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.628629 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.628785 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.628786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.628963 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:25.629013 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.629014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.629690 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.629691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.630221 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.630223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.630642 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.630643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.630686 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.631147 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:25.631152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:25.633219 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260620 21:59:25.633628 16274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:25.628564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:39759 (local address 127.6.241.190:42295)
0620 21:59:25.628633 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.628635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.628646 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:59:25.628680 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.628681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.628700 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.628778 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:25.629091 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.629609 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.630331 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.630537 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.630659 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.633331 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:39759: BlockingRecv error: recv got EOF from 127.6.241.130:39759 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 21:59:25 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:25 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:25.778077 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.773288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:49013)
0620 21:59:25.773439 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.773449 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.773516 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.773749 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.773750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.773913 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:25.773970 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.773971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.774697 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.774698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.775223 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.775224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.775639 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.775640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.775681 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.776162 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:25.776167 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:25.778014 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260620 21:59:25.778236 16274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:25.773368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:49013 (local address 127.6.241.190:42295)
0620 21:59:25.773448 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.773450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.773549 (+ 99us) server_negotiation.cc:407] Connection header received
0620 21:59:25.773596 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.773597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.773624 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.773740 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:25.774043 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.774550 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.775365 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.775560 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.775629 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.778114 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:49013: BlockingRecv error: recv got EOF from 127.6.241.131:49013 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 21:59:26 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:26 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:26.507263 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.502018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:33595)
0620 21:59:26.502255 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.502270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.502353 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.502550 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.502551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.502751 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:26.502803 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.502805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.503502 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.503504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.504043 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.504044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.504518 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.504519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.504593 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.505202 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:26.505207 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:26.507163 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":189,"threads_started":1}
W20260620 21:59:26.507323 16277 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:26.502075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:33595 (local address 127.6.241.190:42295)
0620 21:59:26.502229 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.502231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.502354 (+ 123us) server_negotiation.cc:407] Connection header received
0620 21:59:26.502404 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.502405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.502422 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.502493 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:26.502875 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.503380 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.504119 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.504431 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.504517 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.507253 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:33595: BlockingRecv error: recv got EOF from 127.6.241.129:33595 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
Jun 20 21:59:26 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:26 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:26.639304 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.633994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:47373)
0620 21:59:26.634164 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.634175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.634238 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.634465 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.634466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.634714 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:26.634764 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.634766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.635464 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.635465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.636199 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.636202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.636733 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.636734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.636789 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.637358 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:26.637364 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:26.639208 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
W20260620 21:59:26.639382 16277 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:26.634213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:47373 (local address 127.6.241.190:42295)
0620 21:59:26.634302 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.634304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.634314 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:26.634341 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.634341 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.634352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.634418 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:26.634864 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.635367 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.636314 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.636649 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.636732 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.639305 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:47373: BlockingRecv error: recv got EOF from 127.6.241.130:47373 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 21:59:26 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:26 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:26.783677 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.778908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:57891)
0620 21:59:26.779096 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.779107 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.779183 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.779335 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.779336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.779510 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:26.779559 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.779561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.780215 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.780216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.780821 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.780824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.781252 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.781253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.781309 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.781764 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:26.781768 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:26.783616 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":122,"threads_started":1}
W20260620 21:59:26.783851 16277 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:26.778966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:57891 (local address 127.6.241.190:42295)
0620 21:59:26.779048 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.779050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.779206 (+ 156us) server_negotiation.cc:407] Connection header received
0620 21:59:26.779241 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.779242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.779256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.779318 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:26.779637 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.780140 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.780972 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.781156 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.781250 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.783730 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:57891: BlockingRecv error: recv got EOF from 127.6.241.131:57891 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 21:59:27 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:27 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:27.512753 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:27.507848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:43423)
0620 21:59:27.508013 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:27.508028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:27.508101 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:27.508254 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:27.508255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:27.508434 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:27.508484 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:27.508486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.509154 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.509155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:27.509737 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:27.509738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.510105 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.510106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:27.510156 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:27.510661 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:27.510666 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:27.512673 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
W20260620 21:59:27.512943 16282 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:27.507915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:43423 (local address 127.6.241.190:42295)
0620 21:59:27.508078 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:27.508080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:27.508089 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:27.508165 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:27.508165 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:27.508176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:27.508234 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:27.508560 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.509085 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.509814 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.510012 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.510066 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:27.512812 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:43423: BlockingRecv error: recv got EOF from 127.6.241.129:43423 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
Jun 20 21:59:27 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:27 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:27.644614 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:27.639815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:60179)
0620 21:59:27.639971 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:27.639986 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:27.640045 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:27.640160 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:27.640160 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:27.640329 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:27.640380 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:27.640382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.641056 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.641057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:27.641602 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:27.641604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.641953 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.641954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:27.642006 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:27.642497 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:27.642502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:27.644514 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
W20260620 21:59:27.644675 16282 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:27.639912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:60179 (local address 127.6.241.190:42295)
0620 21:59:27.639982 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:27.639984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:27.640032 (+ 48us) server_negotiation.cc:407] Connection header received
0620 21:59:27.640091 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:27.640092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:27.640103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:27.640154 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:27.640471 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.640978 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.641681 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.641867 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.641925 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:27.644604 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:60179: BlockingRecv error: recv got EOF from 127.6.241.130:60179 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 21:59:27 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:27 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:27.788954 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:27.784203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:54585)
0620 21:59:27.784464 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:27.784476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:27.784539 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:27.784671 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:27.784671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:27.784836 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:27.784897 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:27.784899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.785545 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.785545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:27.786070 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:27.786072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.786396 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.786397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:27.786436 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:27.786947 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:27.786952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:27.788867 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":185,"threads_started":1}
W20260620 21:59:27.789042 16282 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:27.784251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:54585 (local address 127.6.241.190:42295)
0620 21:59:27.784377 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:27.784379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:27.784550 (+ 171us) server_negotiation.cc:407] Connection header received
0620 21:59:27.784580 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:27.784581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:27.784594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:27.784651 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:27.784974 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.785476 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.786148 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:27.786326 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:27.786383 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:27.788978 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:54585: BlockingRecv error: recv got EOF from 127.6.241.131:54585 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 21:59:28 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:28 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:28.518489 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:28.513407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:35447)
0620 21:59:28.513606 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:28.513620 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:28.513698 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:28.513879 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:28.513881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:28.514057 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:28.514112 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:28.514114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.514925 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.514926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:28.515537 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:28.515539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.515965 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.515966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:28.516018 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:28.516556 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:28.516560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:28.518427 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
W20260620 21:59:28.518774 16286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:28.513543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:35447 (local address 127.6.241.190:42295)
0620 21:59:28.513693 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:28.513695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:28.513705 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:28.513758 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:28.513759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:28.513769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:28.513831 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:28.514232 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.514848 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.515637 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.515859 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.515925 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:28.518545 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:35447: BlockingRecv error: recv got EOF from 127.6.241.129:35447 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
Jun 20 21:59:28 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:28 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:28.649971 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:28.645267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:37519)
0620 21:59:28.645464 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:28.645480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:28.645544 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:28.645750 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:28.645750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:28.645957 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:28.646016 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:28.646018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.646831 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.646832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:28.647364 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:28.647366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.647698 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.647699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:28.647741 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:28.648191 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:28.648196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:28.649911 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
W20260620 21:59:28.650107 16286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:28.645352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:37519 (local address 127.6.241.190:42295)
0620 21:59:28.645432 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:28.645434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:28.645550 (+ 116us) server_negotiation.cc:407] Connection header received
0620 21:59:28.645633 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:28.645634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:28.645648 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:28.645715 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:28.646098 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.646750 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.647443 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.647629 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.647680 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:28.650020 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:37519: BlockingRecv error: recv got EOF from 127.6.241.130:37519 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 21:59:28 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:28 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:28.794170 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:28.789527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:44893)
0620 21:59:28.789673 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:28.789689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:28.789744 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:28.789884 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:28.789885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:28.790051 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:28.790102 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:28.790103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.790755 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.790756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:28.791272 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:28.791273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.791602 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.791603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:28.791642 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:28.792099 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:28.792104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:28.794056 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
W20260620 21:59:28.794209 16286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:28.789675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:44893 (local address 127.6.241.190:42295)
0620 21:59:28.789747 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:28.789749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:28.789774 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:59:28.789796 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:28.789797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:28.789809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:28.789864 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:28.790169 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.790684 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.791347 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.791531 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.791588 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:28.794140 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:44893: BlockingRecv error: recv got EOF from 127.6.241.131:44893 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 21:59:29 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:29 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:29.524237 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:29.519159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:41805)
0620 21:59:29.519299 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:29.519318 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:29.519372 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:29.519655 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:29.519656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:29.519826 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:29.519876 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:29.519877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.520566 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.520567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:29.521107 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:29.521108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.521498 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.521499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:29.521539 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:29.522127 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:29.522133 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:29.524166 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
W20260620 21:59:29.524371 16290 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:29.519358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:41805 (local address 127.6.241.190:42295)
0620 21:59:29.519504 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:29.519506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:29.519515 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:29.519549 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:29.519550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:29.519566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:29.519636 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:29.519970 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.520483 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.521206 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.521422 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.521483 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:29.524270 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:41805: BlockingRecv error: recv got EOF from 127.6.241.129:41805 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
Jun 20 21:59:29 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:29 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:29.656311 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:29.650722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:42581)
0620 21:59:29.650880 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:29.650894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:29.650957 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:29.651125 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:29.651126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:29.651301 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:29.651378 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:29.651380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.652216 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.652217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:29.652842 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:29.652845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.653243 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.653244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:29.653285 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:29.653854 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:29.653859 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:29.656192 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
W20260620 21:59:29.656420 16290 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:29.650886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:42581 (local address 127.6.241.190:42295)
0620 21:59:29.650941 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:29.650946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:29.650961 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:59:29.651040 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:29.651041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:29.651052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:29.651102 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:29.651462 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.652122 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.652967 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.653164 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.653228 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:29.656328 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:42581: BlockingRecv error: recv got EOF from 127.6.241.130:42581 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 21:59:29 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:29 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:29.800087 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:29.794879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:52917)
0620 21:59:29.795077 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:29.795097 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:29.795175 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:29.795336 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:29.795337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:29.795510 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:29.795559 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:29.795561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.796371 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.796371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:29.796907 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:29.796908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.797387 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.797388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:29.797438 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:29.797920 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:29.797925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:29.800026 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
W20260620 21:59:29.800280 16290 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:29.794952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:52917 (local address 127.6.241.190:42295)
0620 21:59:29.795084 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:29.795086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:29.795174 (+ 88us) server_negotiation.cc:407] Connection header received
0620 21:59:29.795239 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:29.795240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:29.795253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:29.795308 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:29.795628 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.796287 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.797012 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.797303 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.797375 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:29.800198 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:52917: BlockingRecv error: recv got EOF from 127.6.241.131:52917 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 21:59:30 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:30 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:30.534967 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:30.525055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:44447)
0620 21:59:30.525282 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:30.525297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:30.525391 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:30.525574 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:30.525575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:30.525750 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:30.525802 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:30.525804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.526525 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.526525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:30.527168 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:30.527170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.527539 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.527540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:30.527583 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:30.532691 (+ 5108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:30.532698 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:30.534899 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":64,"threads_started":1}
W20260620 21:59:30.535166 16294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:30.525230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:44447 (local address 127.6.241.190:42295)
0620 21:59:30.525395 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:30.525397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:30.525411 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:59:30.525438 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:30.525439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:30.525449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:30.525514 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:30.525908 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.526435 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.527254 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.527461 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.527518 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:30.535049 (+ 7531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:44447: BlockingRecv error: recv got EOF from 127.6.241.129:44447 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
Jun 20 21:59:30 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:30 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:30.662165 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:30.656954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:33167)
0620 21:59:30.657163 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:30.657181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:30.657263 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:30.657541 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:30.657542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:30.657798 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:30.657866 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:30.657869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.658681 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.658682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:30.659228 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:30.659230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.659632 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.659633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:30.659683 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:30.660171 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:30.660175 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:30.662105 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":143,"threads_started":1}
W20260620 21:59:30.662411 16294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:30.657072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:33167 (local address 127.6.241.190:42295)
0620 21:59:30.657155 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:30.657157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:30.657370 (+ 213us) server_negotiation.cc:407] Connection header received
0620 21:59:30.657415 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:30.657416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:30.657433 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:30.657501 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:30.657965 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.658540 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.659348 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.659536 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.659593 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:30.662297 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:33167: BlockingRecv error: recv got EOF from 127.6.241.130:33167 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 21:59:30 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:30 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:30.805974 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:30.800691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:34751)
0620 21:59:30.800858 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:30.800873 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:30.800933 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:30.801098 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:30.801099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:30.801287 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:30.801338 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:30.801340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.802133 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.802134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:30.803034 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:30.803036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.803380 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.803381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:30.803425 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:30.803957 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:30.803962 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:30.805861 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
W20260620 21:59:30.806010 16294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:30.800786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:34751 (local address 127.6.241.190:42295)
0620 21:59:30.800888 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:30.800890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:30.800922 (+ 32us) server_negotiation.cc:407] Connection header received
0620 21:59:30.801000 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:30.801001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:30.801013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:30.801078 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:30.801445 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.801975 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.803110 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:30.803306 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:30.803363 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:30.805943 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:34751: BlockingRecv error: recv got EOF from 127.6.241.131:34751 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 21:59:31 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:31 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:31.541133 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:31.535709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:40291)
0620 21:59:31.535882 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:31.535894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:31.535964 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:31.536108 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:31.536109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:31.536332 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:31.536387 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:31.536388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.537198 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.537199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:31.537756 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:31.537758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.538137 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.538138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:31.538188 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:31.538810 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:31.538818 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:31.541060 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
W20260620 21:59:31.541262 16298 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:31.535775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:40291 (local address 127.6.241.190:42295)
0620 21:59:31.535897 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:31.535900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:31.535947 (+ 47us) server_negotiation.cc:407] Connection header received
0620 21:59:31.536014 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:31.536015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:31.536026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:31.536089 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:31.536465 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.537114 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.537850 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.538049 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.538107 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:31.541175 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:40291: BlockingRecv error: recv got EOF from 127.6.241.129:40291 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
Jun 20 21:59:31 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:31 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:31.667771 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:31.662912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:59377)
0620 21:59:31.663091 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:31.663102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:31.663163 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:31.663306 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:31.663307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:31.663487 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:31.663539 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:31.663541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.664179 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.664180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:31.664721 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:31.664723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.665122 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.665123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:31.665166 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:31.665749 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:31.665754 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:31.667707 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
W20260620 21:59:31.667894 16298 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:31.662993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:59377 (local address 127.6.241.190:42295)
0620 21:59:31.663093 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:31.663095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:31.663165 (+ 70us) server_negotiation.cc:407] Connection header received
0620 21:59:31.663222 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:31.663222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:31.663255 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:31.663299 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:31.663602 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.664099 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.664834 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.665038 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.665108 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:31.667813 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:59377: BlockingRecv error: recv got EOF from 127.6.241.130:59377 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 21:59:31 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:31 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:31.811316 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:31.806586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:47559)
0620 21:59:31.806766 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:31.806781 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:31.806866 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:31.807021 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:31.807022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:31.807214 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:31.807265 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:31.807287 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.807959 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.807960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:31.808547 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:31.808549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.808894 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.808895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:31.808941 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:31.809407 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:31.809412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:31.811229 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
W20260620 21:59:31.811383 16298 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:31.806788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:47559 (local address 127.6.241.190:42295)
0620 21:59:31.806853 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:31.806855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:31.806864 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:31.806933 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:31.806934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:31.806947 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:31.806997 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:31.807351 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.807862 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.808628 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:31.808807 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:31.808860 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:31.811334 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:47559: BlockingRecv error: recv got EOF from 127.6.241.131:47559 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 21:59:32 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:32 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:32.546911 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:32.541750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:48831)
0620 21:59:32.541978 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:32.541991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:32.542053 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:32.542196 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:32.542197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:32.542372 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:32.542425 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:32.542427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.543122 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.543123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:32.543685 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:32.543686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.544026 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.544026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:32.544069 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:32.544647 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:32.544652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:32.546824 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260620 21:59:32.547065 16302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:32.541802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:48831 (local address 127.6.241.190:42295)
0620 21:59:32.541933 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:32.541935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:32.542039 (+ 104us) server_negotiation.cc:407] Connection header received
0620 21:59:32.542110 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:32.542110 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:32.542121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:32.542171 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:32.542505 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.543039 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.543762 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.543954 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.544006 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:32.546958 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:48831: BlockingRecv error: recv got EOF from 127.6.241.129:48831 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
Jun 20 21:59:32 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:32 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:32.673708 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:32.668324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:47575)
0620 21:59:32.668564 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:32.668578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:32.668687 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:32.668886 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:32.668887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:32.669081 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:32.669128 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:32.669130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.669837 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.669839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:32.670385 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:32.670387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.670858 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.670859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:32.670924 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:32.671514 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:32.671559 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:32.673594 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
W20260620 21:59:32.673761 16302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:32.668453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:47575 (local address 127.6.241.190:42295)
0620 21:59:32.668567 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:32.668569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:32.668665 (+ 96us) server_negotiation.cc:407] Connection header received
0620 21:59:32.668783 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:32.668785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:32.668798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:32.668875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:32.669224 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.669735 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.670492 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.670761 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.670857 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:32.673693 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:47575: BlockingRecv error: recv got EOF from 127.6.241.130:47575 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 21:59:32 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:32 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:32.817139 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:32.811875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:39767)
0620 21:59:32.812115 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:32.812128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:32.812234 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:32.812430 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:32.812431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:32.812630 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:32.812675 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:32.812677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.813412 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.813414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:32.813990 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:32.813991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.814416 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.814417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:32.814479 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:32.815130 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:32.815135 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:32.817023 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":141,"threads_started":1}
W20260620 21:59:32.817186 16302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:32.811989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:39767 (local address 127.6.241.190:42295)
0620 21:59:32.812112 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:32.812114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:32.812215 (+ 101us) server_negotiation.cc:407] Connection header received
0620 21:59:32.812328 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:32.812329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:32.812344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:32.812421 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:32.812773 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.813302 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.814094 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.814314 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.814414 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:32.817116 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:39767: BlockingRecv error: recv got EOF from 127.6.241.131:39767 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 21:59:33 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:33 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:33.552136 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:33.547448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:33943)
0620 21:59:33.547612 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:33.547624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:33.547682 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:33.547837 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:33.547837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:33.548017 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:33.548074 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:33.548076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.548769 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.548770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:33.549342 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:33.549344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.549694 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.549695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:33.549748 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:33.550222 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:33.550227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:33.552073 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
W20260620 21:59:33.552260 16306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:33.547574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:33943 (local address 127.6.241.190:42295)
0620 21:59:33.547705 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:33.547707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:33.547716 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:59:33.547740 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:33.547740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:33.547751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:33.547818 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:33.548157 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.548669 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.549429 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.549624 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.549680 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:33.552178 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:33943: BlockingRecv error: recv got EOF from 127.6.241.129:33943 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
Jun 20 21:59:33 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:33 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:33.679847 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:33.674341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:49981)
0620 21:59:33.674524 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:33.674535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:33.674668 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:33.674873 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:33.674874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:33.675067 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:33.675116 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:33.675118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.675858 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.675860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:33.676414 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:33.676416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.676821 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.676822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:33.676884 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:33.677431 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:33.677436 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:33.679736 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":119,"threads_started":1}
W20260620 21:59:33.679901 16306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:33.674519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:49981 (local address 127.6.241.190:42295)
0620 21:59:33.674633 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:33.674635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:33.674646 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:59:33.674754 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:33.674755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:33.674768 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:33.674854 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:33.675249 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.675749 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.676522 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.676725 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.676821 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:33.679834 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:49981: BlockingRecv error: recv got EOF from 127.6.241.130:49981 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 21:59:33 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:33 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:33.822556 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:33.817725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:59081)
0620 21:59:33.817936 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:33.817947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:33.818010 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:33.818158 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:33.818159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:33.818334 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:33.818381 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:33.818383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.819154 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.819155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:33.819704 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:33.819706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.820068 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.820069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:33.820110 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:33.820699 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:33.820704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:33.822479 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
W20260620 21:59:33.822703 16306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:33.817868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:59081 (local address 127.6.241.190:42295)
0620 21:59:33.817918 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:33.817920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:33.817993 (+ 73us) server_negotiation.cc:407] Connection header received
0620 21:59:33.818057 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:33.818058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:33.818070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:33.818130 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:33.818493 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.819073 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.819793 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:33.819996 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:33.820051 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:33.822591 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:59081: BlockingRecv error: recv got EOF from 127.6.241.131:59081 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 21:59:34 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:34 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:34.558413 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:34.552839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:49917)
0620 21:59:34.553023 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:34.553041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:34.553100 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:34.553324 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:34.553325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:34.553524 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:34.553572 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:34.553574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.554412 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.554413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:34.555009 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:34.555011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.555517 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.555519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:34.555585 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:34.556288 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:34.556294 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:34.558288 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":123,"threads_started":1}
W20260620 21:59:34.558461 16310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:34.553016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:49917 (local address 127.6.241.190:42295)
0620 21:59:34.553172 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:34.553174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:34.553184 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:34.553212 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:34.553212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:34.553224 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:34.553300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:34.553700 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.554266 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.555128 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.555337 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.555409 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:34.558386 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:49917: BlockingRecv error: recv got EOF from 127.6.241.129:49917 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
Jun 20 21:59:34 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:34 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:34.685698 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:34.680513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:33307)
0620 21:59:34.680724 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:34.680741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:34.680821 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:34.680977 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:34.680978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:34.681152 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:34.681203 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:34.681205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.681946 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.681947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:34.682484 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:34.682486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.682971 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.682972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:34.683019 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:34.683504 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:34.683509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:34.685599 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
W20260620 21:59:34.685768 16310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:34.680705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:33307 (local address 127.6.241.190:42295)
0620 21:59:34.680783 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:34.680785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:34.680797 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:59:34.680889 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:34.680890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:34.680902 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:34.680944 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:34.681321 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.681845 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.682580 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.682894 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.682958 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:34.685704 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:33307: BlockingRecv error: recv got EOF from 127.6.241.130:33307 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 21:59:34 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:34 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:34.828014 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:34.823227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:55259)
0620 21:59:34.823425 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:34.823437 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:34.823502 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:34.823645 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:34.823646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:34.823834 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:34.823883 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:34.823885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.824622 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.824623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:34.825172 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:34.825174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.825565 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.825566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:34.825610 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:34.826087 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:34.826092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:34.827951 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
W20260620 21:59:34.828142 16310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:34.823357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:55259 (local address 127.6.241.190:42295)
0620 21:59:34.823438 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:34.823440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:34.823538 (+ 98us) server_negotiation.cc:407] Connection header received
0620 21:59:34.823561 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:34.823562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:34.823574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:34.823616 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:34.823991 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.824522 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.825267 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.825490 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.825551 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:34.828054 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:55259: BlockingRecv error: recv got EOF from 127.6.241.131:55259 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 21:59:35 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:35 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:35.568567 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:35.559174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:46217)
0620 21:59:35.559359 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:35.559373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:35.559440 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:35.559612 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:35.559613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:35.559809 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:35.559860 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:35.559862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.560553 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.560554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:35.561089 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:35.561090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.561450 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.561451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:35.561495 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:35.566330 (+ 4835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:35.566338 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:35.568503 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":132,"threads_started":1}
W20260620 21:59:35.568733 16314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:35.559277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:46217 (local address 127.6.241.190:42295)
0620 21:59:35.559457 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:35.559459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:35.559491 (+ 32us) server_negotiation.cc:407] Connection header received
0620 21:59:35.559515 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:35.559516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:35.559526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:35.559586 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:35.559952 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.560475 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.561174 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.561373 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.561428 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:35.568627 (+ 7199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:46217: BlockingRecv error: recv got EOF from 127.6.241.129:46217 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
Jun 20 21:59:35 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:35 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:35.691473 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:35.686454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:55169)
0620 21:59:35.686636 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:35.686647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:35.686711 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:35.686839 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:35.686840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:35.687024 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:35.687076 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:35.687077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.687906 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.687908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:35.688552 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:35.688554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.688911 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.688912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:35.688952 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:35.689409 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:35.689414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:35.691401 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
W20260620 21:59:35.691581 16314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:35.686540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:55169 (local address 127.6.241.190:42295)
0620 21:59:35.686650 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:35.686658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:35.686715 (+ 57us) server_negotiation.cc:407] Connection header received
0620 21:59:35.686763 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:35.686764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:35.686775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:35.686834 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:35.687142 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.687842 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.688641 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.688827 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.688885 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:35.691507 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:55169: BlockingRecv error: recv got EOF from 127.6.241.130:55169 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 21:59:35 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:35 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.131@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:35.833986 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:35.828724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.131:59831)
0620 21:59:35.828903 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:35.828916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:35.828993 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:35.829144 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:35.829144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:35.829322 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:35.829370 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:35.829372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.830142 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.830143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:35.830728 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:35.830731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.831130 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.831131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:35.831172 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:35.831681 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:35.831686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:35.833925 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":120,"threads_started":1}
W20260620 21:59:35.834115 16314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:35.828816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:59831 (local address 127.6.241.190:42295)
0620 21:59:35.828920 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:35.828922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:35.829004 (+ 82us) server_negotiation.cc:407] Connection header received
0620 21:59:35.829036 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:35.829037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:35.829050 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:35.829115 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:35.829489 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.830018 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.830849 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:35.831043 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:35.831111 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:35.834029 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.131:59831: BlockingRecv error: recv got EOF from 127.6.241.131:59831 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 21:59:36 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:36 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.129@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:36.574873 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:36.569201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.129:34137)
0620 21:59:36.569400 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:36.569414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:36.569482 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:36.569666 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:36.569667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:36.569850 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:36.569903 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:36.569905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.570652 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.570653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:36.571436 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:36.571439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.571907 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.571908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:36.571982 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:36.572621 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:36.572639 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:36.574755 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
W20260620 21:59:36.574958 16318 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:36.569262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:34137 (local address 127.6.241.190:42295)
0620 21:59:36.569496 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:36.569499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:36.569510 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:59:36.569533 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:36.569534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:36.569545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:36.569620 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:36.569992 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.570509 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.571559 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.571794 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.571903 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:36.574875 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.129:34137: BlockingRecv error: recv got EOF from 127.6.241.129:34137 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":127,"threads_started":1}
Jun 20 21:59:36 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:59:36 dist-test-slave-gl2g krb5kdc[15259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.241.130@KRBTEST.COM for oryx/127.6.241.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:59:36.697887 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:36.692174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42295 (local address 127.6.241.130:60363)
0620 21:59:36.692341 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:36.692353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:36.692447 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:36.692601 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:36.692602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:36.692813 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:36.692865 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:36.692867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.693685 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.693686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:36.694293 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:36.694296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.694828 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.694829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:36.694899 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:36.695543 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:36.695549 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:36.697814 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.241.190:42295: Server oryx/127.6.241.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
W20260620 21:59:36.698048 16318 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:36.692224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:60363 (local address 127.6.241.190:42295)
0620 21:59:36.692288 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:36.692291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:36.692424 (+ 133us) server_negotiation.cc:407] Connection header received
0620 21:59:36.692522 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:36.692522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:36.692536 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:36.692596 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:36.692963 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.693562 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.694395 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.694732 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.694829 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:36.697958 (+ 3129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.241.130:60363: BlockingRecv error: recv got EOF from 127.6.241.130:60363 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:59:36.747274 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 15842
I20260620 21:59:36.752317 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 15994
I20260620 21:59:36.757102 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 16126
I20260620 21:59:36.761718 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 15772
2026-06-20T21:59:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17494 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T21:59:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:36Z Disabled control of system clock
I20260620 21:59:36.793239 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33977
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:33977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 21:59:36.883137 16323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:36.883381 16323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:36.883404 16323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:36.884645 16323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:36.884684 16323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:36.884698 16323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:36.884712 16323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:36.884724 16323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:36.886096 16323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:33977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:36.886327 16323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:36.886629 16323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:36.892956 16323 server_base.cc:1034] running on GCE node
W20260620 21:59:36.892912 16331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:36.892940 16329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:36.892901 16328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:36.893554 16323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:36.893893 16323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:36.895131 16323 hybrid_clock.cc:648] HybridClock initialized: now 1781992776895127 us; error 53 us; skew 500 ppm
I20260620 21:59:36.897181 16323 webserver.cc:460] Webserver started at http://127.6.241.190:35683/ using document root <none> and password file <none>
I20260620 21:59:36.897799 16323 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:36.897903 16323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:36.898092 16323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:36.899565 16323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "2f558a7fede5466e861c3ba5974a2235"
format_stamp: "Formatted at 2026-06-20 21:59:36 on dist-test-slave-gl2g"
server_key: "d7f6dd7cce790ff14a2ba78965b64ea2"
server_key_iv: "3b8127e6d3445a5ac79372e6b2f9845b"
server_key_version: "encryptionkey@0"
I20260620 21:59:36.900010 16323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "2f558a7fede5466e861c3ba5974a2235"
format_stamp: "Formatted at 2026-06-20 21:59:36 on dist-test-slave-gl2g"
server_key: "d7f6dd7cce790ff14a2ba78965b64ea2"
server_key_iv: "3b8127e6d3445a5ac79372e6b2f9845b"
server_key_version: "encryptionkey@0"
I20260620 21:59:36.901679 16323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 21:59:36.902755 16337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:36.903074 16323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 21:59:36.903239 16323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "2f558a7fede5466e861c3ba5974a2235"
format_stamp: "Formatted at 2026-06-20 21:59:36 on dist-test-slave-gl2g"
server_key: "d7f6dd7cce790ff14a2ba78965b64ea2"
server_key_iv: "3b8127e6d3445a5ac79372e6b2f9845b"
server_key_version: "encryptionkey@0"
I20260620 21:59:36.903337 16323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:36.926169 16323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:36.926576 16323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:36.926790 16323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:36.931023 16323 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:33977
I20260620 21:59:36.931061 16389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:33977 every 8 connection(s)
I20260620 21:59:36.931456 16323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:36.932186 16390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:36.935114 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235: Bootstrap starting.
I20260620 21:59:36.935956 16390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:36.936404 16390 log.cc:824] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:36.937283 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235: No bootstrap required, opened a new log
I20260620 21:59:36.939167 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 16323
I20260620 21:59:36.939304 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:36.939500 7110 external_mini_cluster.cc:1428] Setting key fddcf756e45325db60018da34f9c6488
I20260620 21:59:36.939409 16390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f558a7fede5466e861c3ba5974a2235" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33977 } }
I20260620 21:59:36.939621 16390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:36.939693 16390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f558a7fede5466e861c3ba5974a2235, State: Initialized, Role: FOLLOWER
I20260620 21:59:36.939862 16390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f558a7fede5466e861c3ba5974a2235" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33977 } }
I20260620 21:59:36.939970 16390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:36.940045 16390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:36.940126 16390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:36.940975 16390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f558a7fede5466e861c3ba5974a2235" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33977 } }
I20260620 21:59:36.941099 16390 leader_election.cc:302] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f558a7fede5466e861c3ba5974a2235; no voters:
I20260620 21:59:36.941283 16390 leader_election.cc:288] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:36.941378 16395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:36.941573 16395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [term 1 LEADER]: Becoming Leader. State: Replica: 2f558a7fede5466e861c3ba5974a2235, State: Running, Role: LEADER
I20260620 21:59:36.941598 16390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:36.941716 16395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f558a7fede5466e861c3ba5974a2235" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33977 } }
I20260620 21:59:36.942116 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f558a7fede5466e861c3ba5974a2235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f558a7fede5466e861c3ba5974a2235" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33977 } } }
I20260620 21:59:36.942191 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:36.942437 16397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f558a7fede5466e861c3ba5974a2235. Latest consensus state: current_term: 1 leader_uuid: "2f558a7fede5466e861c3ba5974a2235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f558a7fede5466e861c3ba5974a2235" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33977 } } }
I20260620 21:59:36.942765 16400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:36.942786 16397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:36.943368 16400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:36.945187 16400 catalog_manager.cc:1347] Generated new cluster ID: ed862f056fac412a8427ab37640e8c24
I20260620 21:59:36.945248 16400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:36.946955 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:36.940063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.6.241.190:33977)
0620 21:59:36.940344 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:36.940354 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:36.940389 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:59:36.940521 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:36.940528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:36.940616 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:36.940802 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:36.941034 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.943015 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.944178 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:36.945042 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:36.945538 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:36.946024 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:36.946028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:36.946029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:36.946234 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:36.946272 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:36.946286 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:36.946584 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:36.946646 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 21:59:36.946728 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":110,"threads_started":1}
I20260620 21:59:36.964167 16400 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:36.964995 16400 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:36.971801 16400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f558a7fede5466e861c3ba5974a2235: Generated new TSK 0
I20260620 21:59:36.972055 16400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:36.977295 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33977
--builtin_ntp_servers=127.6.241.148:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 21:59:37.076398 16414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:37.076681 16414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:37.076719 16414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:37.078009 16414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:37.078086 16414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:37.078132 16414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:37.079607 16414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:37.079898 16414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:37.080220 16414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:37.085538 16422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.085548 16419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.085538 16420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.086006 16414 server_base.cc:1034] running on GCE node
I20260620 21:59:37.086244 16414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:37.086539 16414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:37.087719 16414 hybrid_clock.cc:648] HybridClock initialized: now 1781992777087706 us; error 31 us; skew 500 ppm
I20260620 21:59:37.089298 16414 webserver.cc:460] Webserver started at http://127.6.241.129:36857/ using document root <none> and password file <none>
I20260620 21:59:37.089560 16414 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:37.089632 16414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:37.089792 16414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:37.091288 16414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "8ec90c3a765c4cf4a56ba63b138692a3"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "f8d0f616225dfcd48696e85c9ccb231b"
server_key_iv: "467973eaf55a33b7ac6e440c1df58d4f"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.091727 16414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "8ec90c3a765c4cf4a56ba63b138692a3"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "f8d0f616225dfcd48696e85c9ccb231b"
server_key_iv: "467973eaf55a33b7ac6e440c1df58d4f"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.093200 16414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:37.094044 16428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.094226 16414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:37.094329 16414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "8ec90c3a765c4cf4a56ba63b138692a3"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "f8d0f616225dfcd48696e85c9ccb231b"
server_key_iv: "467973eaf55a33b7ac6e440c1df58d4f"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.094419 16414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:37.104928 16414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:37.105285 16414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:37.105450 16414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:37.105722 16414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:37.106092 16414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:37.106151 16414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.106189 16414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:37.106249 16414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.112737 16414 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:33087
I20260620 21:59:37.112805 16541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:33087 every 8 connection(s)
I20260620 21:59:37.113189 16414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:37.113667 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 16414
I20260620 21:59:37.113792 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:37.114004 7110 external_mini_cluster.cc:1428] Setting key d2fadc3c0877d6feacbcc276b6e10931
I20260620 21:59:37.115111 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33977
--builtin_ntp_servers=127.6.241.148:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 21:59:37.119634 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.114171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:44855 (local address 127.6.241.190:33977)
0620 21:59:37.114233 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.114236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.114478 (+ 242us) server_negotiation.cc:407] Connection header received
0620 21:59:37.115129 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.115131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.115145 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.115198 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.115962 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.116789 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.117841 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.118175 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.118259 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:37.119207 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:37.119209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:37.119209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:37.119215 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:37.119236 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:37.119242 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:37.119316 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:37.119464 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 21:59:37.119509 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 21:59:37.119788 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.114138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33977 (local address 127.6.241.129:44855)
0620 21:59:37.114341 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:37.114388 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:37.114994 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:37.115284 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:37.115287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:37.115296 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:37.115847 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:37.115852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.116914 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.116916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:37.117742 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:37.117744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.118268 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.118269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:37.118395 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:37.119042 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:37.119043 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:37.119075 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:37.119102 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:37.119356 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:37.119363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:37.119366 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:37.119441 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 21:59:37.119540 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260620 21:59:37.120621 16542 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33977
I20260620 21:59:37.120745 16542 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:37.120973 16542 heartbeater.cc:511] Master 127.6.241.190:33977 requested a full tablet report, sending...
I20260620 21:59:37.121599 16354 ts_manager.cc:194] Registered new tserver with Master: 8ec90c3a765c4cf4a56ba63b138692a3 (127.6.241.129:33087)
I20260620 21:59:37.122257 16354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.129:44855
W20260620 21:59:37.202154 16545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:37.202385 16545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:37.202409 16545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:37.203709 16545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:37.203765 16545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:37.203796 16545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:37.205247 16545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:37.205528 16545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:37.205911 16545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:37.211383 16551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.211385 16550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.211386 16553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.211791 16545 server_base.cc:1034] running on GCE node
I20260620 21:59:37.211989 16545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:37.212222 16545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:37.213439 16545 hybrid_clock.cc:648] HybridClock initialized: now 1781992777213424 us; error 27 us; skew 500 ppm
I20260620 21:59:37.214941 16545 webserver.cc:460] Webserver started at http://127.6.241.130:46343/ using document root <none> and password file <none>
I20260620 21:59:37.215204 16545 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:37.215282 16545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:37.215456 16545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:37.216629 16545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "efdd21cd793244038b915cefaac20b4a"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "dfbc4fcc7e519e27720275fb8c64bb55"
server_key_iv: "65f2e1a266fb59f8e85272ec9b19fdbd"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.217054 16545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "efdd21cd793244038b915cefaac20b4a"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "dfbc4fcc7e519e27720275fb8c64bb55"
server_key_iv: "65f2e1a266fb59f8e85272ec9b19fdbd"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.218516 16545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:37.219435 16559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.219656 16545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:37.219780 16545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "efdd21cd793244038b915cefaac20b4a"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "dfbc4fcc7e519e27720275fb8c64bb55"
server_key_iv: "65f2e1a266fb59f8e85272ec9b19fdbd"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.219862 16545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:37.233879 16545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:37.234261 16545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:37.234432 16545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:37.234783 16545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:37.235149 16545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:37.235208 16545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.235250 16545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:37.235280 16545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.241609 16545 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:41809
I20260620 21:59:37.241703 16672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:41809 every 8 connection(s)
I20260620 21:59:37.242146 16545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:37.247794 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.243045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:41497 (local address 127.6.241.190:33977)
0620 21:59:37.243095 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.243097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.243350 (+ 253us) server_negotiation.cc:407] Connection header received
0620 21:59:37.243916 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.243917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.243929 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.243987 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.244718 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.245221 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.246281 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.246502 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.246563 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:37.247548 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:37.247549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:37.247549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:37.247554 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:37.247567 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:37.247571 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:37.247615 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:37.247682 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 21:59:37.247708 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 21:59:37.247974 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.243059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33977 (local address 127.6.241.130:41497)
0620 21:59:37.243231 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:37.243266 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:37.243782 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:37.244068 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:37.244072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:37.244081 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:37.244601 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:37.244606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.245344 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.245345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:37.246180 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:37.246182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.246612 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.246613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:37.246739 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:37.247409 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:37.247409 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:37.247442 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:37.247467 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:37.247616 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:37.247618 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:37.247621 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:37.247670 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 21:59:37.247760 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260620 21:59:37.248513 16673 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33977
I20260620 21:59:37.248631 16673 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:37.248824 16673 heartbeater.cc:511] Master 127.6.241.190:33977 requested a full tablet report, sending...
I20260620 21:59:37.249251 16354 ts_manager.cc:194] Registered new tserver with Master: efdd21cd793244038b915cefaac20b4a (127.6.241.130:41809)
I20260620 21:59:37.249645 16354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.130:41497
I20260620 21:59:37.250820 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 16545
I20260620 21:59:37.250895 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:37.251051 7110 external_mini_cluster.cc:1428] Setting key f59665e6547bb40d58285fd1a64e917f
I20260620 21:59:37.251941 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33977
--builtin_ntp_servers=127.6.241.148:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 21:59:37.336841 16676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:37.337121 16676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:37.337157 16676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:37.338367 16676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:37.338485 16676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:37.338532 16676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:37.340021 16676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:37.340322 16676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:37.340663 16676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:37.346095 16681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.346119 16682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.346125 16676 server_base.cc:1034] running on GCE node
W20260620 21:59:37.346104 16684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.346640 16676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:37.346939 16676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:37.348110 16676 hybrid_clock.cc:648] HybridClock initialized: now 1781992777348111 us; error 41 us; skew 500 ppm
I20260620 21:59:37.349488 16676 webserver.cc:460] Webserver started at http://127.6.241.131:36823/ using document root <none> and password file <none>
I20260620 21:59:37.349772 16676 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:37.349849 16676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:37.350021 16676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:37.351233 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "57a88b768d2e4a31991032065863d05f"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "7b5e1cb31abcb75c82ea4d6521f0cc72"
server_key_iv: "c6c13c9a984ca6ab9cd1383f0c861b02"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.351624 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "57a88b768d2e4a31991032065863d05f"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "7b5e1cb31abcb75c82ea4d6521f0cc72"
server_key_iv: "c6c13c9a984ca6ab9cd1383f0c861b02"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.353049 16676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:37.353822 16690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.354125 16676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:37.354225 16676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "57a88b768d2e4a31991032065863d05f"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "7b5e1cb31abcb75c82ea4d6521f0cc72"
server_key_iv: "c6c13c9a984ca6ab9cd1383f0c861b02"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.354303 16676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:37.383811 16676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:37.384230 16676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:37.384419 16676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:37.384691 16676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:37.385084 16676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:37.385146 16676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.385190 16676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:37.385221 16676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.391355 16676 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:41599
I20260620 21:59:37.391415 16803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:41599 every 8 connection(s)
I20260620 21:59:37.391889 16676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:37.397364 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 16676
I20260620 21:59:37.397401 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.392755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:37501 (local address 127.6.241.190:33977)
0620 21:59:37.392805 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.392807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.393070 (+ 263us) server_negotiation.cc:407] Connection header received
0620 21:59:37.393566 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.393567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.393577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.393646 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.394311 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.394900 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.395869 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.396062 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.396114 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:37.397051 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:37.397052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:37.397052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:37.397057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:37.397073 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:37.397078 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:37.397166 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:37.397248 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 21:59:37.397289 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:59:37.397471 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:37.397547 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.392713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33977 (local address 127.6.241.131:37501)
0620 21:59:37.392929 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:37.392972 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:37.393479 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:37.393707 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:37.393711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:37.393720 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:37.394230 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:37.394235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.394983 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.394984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:37.395787 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:37.395789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.396141 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.396142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:37.396274 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:37.396909 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:37.396909 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:37.396940 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:37.396966 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:37.397165 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:37.397169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:37.397172 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:37.397226 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 21:59:37.397315 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260620 21:59:37.397722 7110 external_mini_cluster.cc:1428] Setting key 5174369930969d76a8c0674f0bdae658
I20260620 21:59:37.398401 16804 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33977
I20260620 21:59:37.398532 16804 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:37.398797 16804 heartbeater.cc:511] Master 127.6.241.190:33977 requested a full tablet report, sending...
I20260620 21:59:37.399184 16353 ts_manager.cc:194] Registered new tserver with Master: 57a88b768d2e4a31991032065863d05f (127.6.241.131:41599)
I20260620 21:59:37.399613 16353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.131:37501
I20260620 21:59:37.410053 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:59:37.412286 16394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:37.411610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.6.241.190:33977)
0620 21:59:37.411704 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.411707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.411819 (+ 112us) server_negotiation.cc:407] Connection header received
0620 21:59:37.411904 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.411905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.411931 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.411999 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.412154 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46400: BlockingRecv error: recv got EOF from 127.0.0.1:46400 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:59:37.412317 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 16414
I20260620 21:59:37.417142 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 16545
I20260620 21:59:37.421618 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 16676
I20260620 21:59:37.426165 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 16323
2026-06-20T21:59:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (661 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T21:59:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:37Z Disabled control of system clock
I20260620 21:59:37.453852 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40657
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:40657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:59:37.536922 16815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:37.537170 16815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:37.537194 16815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:37.538450 16815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:37.538497 16815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:37.538511 16815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:37.538523 16815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:37.538535 16815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:37.539961 16815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:40657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:37.540189 16815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:37.540454 16815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:37.545670 16820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.545668 16823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.545850 16815 server_base.cc:1034] running on GCE node
W20260620 21:59:37.545686 16821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.546262 16815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:37.546543 16815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:37.547696 16815 hybrid_clock.cc:648] HybridClock initialized: now 1781992777547681 us; error 29 us; skew 500 ppm
I20260620 21:59:37.549183 16815 webserver.cc:460] Webserver started at http://127.6.241.190:45233/ using document root <none> and password file <none>
I20260620 21:59:37.549454 16815 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:37.549531 16815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:37.549692 16815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:37.550910 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "518d7767fc70189b251ba7dfa98cd615"
server_key_iv: "71e3d8ba6cc0e3745dd4283491a5610a"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.551294 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "518d7767fc70189b251ba7dfa98cd615"
server_key_iv: "71e3d8ba6cc0e3745dd4283491a5610a"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.552826 16815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:37.553656 16829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.553846 16815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:37.553964 16815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "518d7767fc70189b251ba7dfa98cd615"
server_key_iv: "71e3d8ba6cc0e3745dd4283491a5610a"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.554049 16815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:37.557699 16815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:37.558017 16815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:37.558168 16815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:37.562407 16815 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:40657
I20260620 21:59:37.562441 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:40657 every 8 connection(s)
I20260620 21:59:37.562901 16815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:37.563612 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:37.566495 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74: Bootstrap starting.
I20260620 21:59:37.567293 16882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:37.567613 16882 log.cc:824] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:37.568392 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74: No bootstrap required, opened a new log
I20260620 21:59:37.568670 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 16815
I20260620 21:59:37.568783 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:37.568993 7110 external_mini_cluster.cc:1428] Setting key 7ba75d4dd65a32b10f318df583a6fc3f
I20260620 21:59:37.570530 16882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40657 } }
I20260620 21:59:37.570819 16882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:37.570869 16882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcefbe335ba4dcabf3c3c27d3a99f74, State: Initialized, Role: FOLLOWER
I20260620 21:59:37.571048 16882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40657 } }
I20260620 21:59:37.571136 16882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:37.571161 16882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:37.571228 16882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:37.571883 16882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40657 } }
I20260620 21:59:37.572014 16882 leader_election.cc:302] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bcefbe335ba4dcabf3c3c27d3a99f74; no voters:
I20260620 21:59:37.572206 16882 leader_election.cc:288] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:37.572291 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.569626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.6.241.190:40657)
0620 21:59:37.569888 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.569897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.569924 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:59:37.570516 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.570524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.570570 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.570794 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.571705 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:37.571709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:37.571709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:37.571712 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:37.571743 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:37.571754 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:37.571837 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:37.571910 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 21:59:37.571977 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":134,"threads_started":1}
I20260620 21:59:37.572649 16887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:37.572794 16887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [term 1 LEADER]: Becoming Leader. State: Replica: 5bcefbe335ba4dcabf3c3c27d3a99f74, State: Running, Role: LEADER
I20260620 21:59:37.572903 16887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40657 } }
I20260620 21:59:37.572993 16882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:37.573225 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bcefbe335ba4dcabf3c3c27d3a99f74. Latest consensus state: current_term: 1 leader_uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40657 } } }
I20260620 21:59:37.573439 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:37.573239 16888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcefbe335ba4dcabf3c3c27d3a99f74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40657 } } }
I20260620 21:59:37.573693 16888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:37.573956 16894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:37.574623 16894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:37.576440 16894 catalog_manager.cc:1347] Generated new cluster ID: 6931c77a402845e5824bee0f8988dacb
I20260620 21:59:37.576503 16894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:37.588301 16894 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:37.588515 16894 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:37.593335 16894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bcefbe335ba4dcabf3c3c27d3a99f74: Generated new TSK 0
I20260620 21:59:37.593582 16894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:37.602487 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40657
--builtin_ntp_servers=127.6.241.148:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:59:37.710135 16906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:37.710371 16906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:37.710397 16906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:37.711777 16906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:37.711835 16906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:37.711864 16906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:37.713338 16906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:37.713634 16906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:37.713973 16906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:37.719296 16912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.719306 16911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.719305 16914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.719895 16906 server_base.cc:1034] running on GCE node
I20260620 21:59:37.720117 16906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:37.720443 16906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:37.721611 16906 hybrid_clock.cc:648] HybridClock initialized: now 1781992777721588 us; error 43 us; skew 500 ppm
I20260620 21:59:37.723028 16906 webserver.cc:460] Webserver started at http://127.6.241.129:42489/ using document root <none> and password file <none>
I20260620 21:59:37.723297 16906 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:37.723379 16906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:37.723538 16906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:37.724692 16906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "436e280d09454e4aafc773bc09baffaa"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "c8c492e643f10ed99e3cfb38dcaa988a"
server_key_iv: "6289d30889c9a76aef74b9d14ddcf86b"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.725099 16906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "436e280d09454e4aafc773bc09baffaa"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "c8c492e643f10ed99e3cfb38dcaa988a"
server_key_iv: "6289d30889c9a76aef74b9d14ddcf86b"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.726495 16906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:37.727365 16920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.727625 16906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:37.727759 16906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "436e280d09454e4aafc773bc09baffaa"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "c8c492e643f10ed99e3cfb38dcaa988a"
server_key_iv: "6289d30889c9a76aef74b9d14ddcf86b"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.727849 16906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:37.731534 16906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:37.731844 16906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:37.731997 16906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:37.732235 16906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:37.732615 16906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:37.732671 16906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.732713 16906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:37.732740 16906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.739818 16906 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:37225
I20260620 21:59:37.739919 17033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:37225 every 8 connection(s)
I20260620 21:59:37.740314 16906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:37.743762 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.741099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:60327 (local address 127.6.241.190:40657)
0620 21:59:37.741146 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.741148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.741454 (+ 306us) server_negotiation.cc:407] Connection header received
0620 21:59:37.741998 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.741999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.742007 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.742066 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.743458 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:37.743460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:37.743460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:37.743465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:37.743482 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:37.743487 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:37.743568 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:37.743668 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 21:59:37.743687 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 21:59:37.744007 17036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.741105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40657 (local address 127.6.241.129:60327)
0620 21:59:37.741306 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:37.741350 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:37.741879 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:37.742158 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:37.742162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:37.742170 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:37.743252 (+ 1082us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:37.743252 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:37.743284 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:37.743345 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:37.743566 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:37.743573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:37.743577 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:37.743679 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 21:59:37.743748 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260620 21:59:37.744576 17034 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40657
I20260620 21:59:37.744663 17034 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:37.744846 17034 heartbeater.cc:511] Master 127.6.241.190:40657 requested a full tablet report, sending...
I20260620 21:59:37.745472 16846 ts_manager.cc:194] Registered new tserver with Master: 436e280d09454e4aafc773bc09baffaa (127.6.241.129:37225)
I20260620 21:59:37.750437 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 16906
I20260620 21:59:37.750586 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:37.750810 7110 external_mini_cluster.cc:1428] Setting key e2eeb8cc69db24f3b416d112f680b2a0
I20260620 21:59:37.751947 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40657
--builtin_ntp_servers=127.6.241.148:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:59:37.837200 17037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:37.837436 17037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:37.837463 17037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:37.838868 17037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:37.838923 17037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:37.838953 17037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:37.840446 17037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:37.840749 17037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:37.841089 17037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:37.846508 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.846549 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.846576 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.846805 17037 server_base.cc:1034] running on GCE node
I20260620 21:59:37.847112 17037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:37.847400 17037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:37.848582 17037 hybrid_clock.cc:648] HybridClock initialized: now 1781992777848554 us; error 45 us; skew 500 ppm
I20260620 21:59:37.850092 17037 webserver.cc:460] Webserver started at http://127.6.241.130:39743/ using document root <none> and password file <none>
I20260620 21:59:37.850366 17037 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:37.850467 17037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:37.850689 17037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:37.851861 17037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "019c7a121ca94ab79f62376c8ba1e1a7"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "3e6b6639ebb59d2bdf290e9e69e48b61"
server_key_iv: "8e122bc3574c9a296575de2e140d5489"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.852259 17037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "019c7a121ca94ab79f62376c8ba1e1a7"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "3e6b6639ebb59d2bdf290e9e69e48b61"
server_key_iv: "8e122bc3574c9a296575de2e140d5489"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.853737 17037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 21:59:37.854576 17051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.854825 17037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:37.854947 17037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "019c7a121ca94ab79f62376c8ba1e1a7"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "3e6b6639ebb59d2bdf290e9e69e48b61"
server_key_iv: "8e122bc3574c9a296575de2e140d5489"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.855044 17037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:37.858768 17037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:37.859105 17037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:37.859265 17037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:37.859485 17037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:37.859833 17037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:37.859905 17037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.859951 17037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:37.859977 17037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.866181 17037 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:32797
I20260620 21:59:37.866227 17164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:32797 every 8 connection(s)
I20260620 21:59:37.866679 17037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:37.866999 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 17037
I20260620 21:59:37.867100 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:37.867311 7110 external_mini_cluster.cc:1428] Setting key 14414c13c19fb701f50324b443cea14b
I20260620 21:59:37.868311 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40657
--builtin_ntp_servers=127.6.241.148:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 21:59:37.870163 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.867506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:39277 (local address 127.6.241.190:40657)
0620 21:59:37.867593 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.867595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.867818 (+ 223us) server_negotiation.cc:407] Connection header received
0620 21:59:37.868410 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.868411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.868418 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.868497 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.869904 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:37.869906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:37.869906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:37.869910 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:37.869927 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:37.869931 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:37.869982 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:37.870091 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 21:59:37.870103 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:59:37.870432 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.867486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40657 (local address 127.6.241.130:39277)
0620 21:59:37.867678 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:37.867720 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:37.868294 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:37.868561 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:37.868565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:37.868574 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:37.869718 (+ 1144us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:37.869719 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:37.869763 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:37.869805 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:37.870005 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:37.870009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:37.870013 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:37.870095 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:59:37.870168 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260620 21:59:37.871050 17165 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40657
I20260620 21:59:37.871160 17165 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:37.871325 17165 heartbeater.cc:511] Master 127.6.241.190:40657 requested a full tablet report, sending...
I20260620 21:59:37.871687 16846 ts_manager.cc:194] Registered new tserver with Master: 019c7a121ca94ab79f62376c8ba1e1a7 (127.6.241.130:32797)
W20260620 21:59:37.954203 17168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:37.954490 17168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:37.954532 17168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:37.955832 17168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:37.955925 17168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:37.955974 17168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:37.957706 17168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:37.958037 17168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:37.958392 17168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:37.964018 17174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.964077 17168 server_base.cc:1034] running on GCE node
W20260620 21:59:37.964025 17173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:37.964033 17176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:37.964421 17168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:37.964695 17168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:37.965844 17168 hybrid_clock.cc:648] HybridClock initialized: now 1781992777965798 us; error 64 us; skew 500 ppm
I20260620 21:59:37.967319 17168 webserver.cc:460] Webserver started at http://127.6.241.131:40417/ using document root <none> and password file <none>
I20260620 21:59:37.967623 17168 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:37.967698 17168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:37.967850 17168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:37.968994 17168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "4e59f694152f452f8d8a9b560c52e258"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "d1e14eab52128e4d61cd9f91cb3e65c3"
server_key_iv: "103e98eb7d41257693a0c276189e66be"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.969388 17168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "4e59f694152f452f8d8a9b560c52e258"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "d1e14eab52128e4d61cd9f91cb3e65c3"
server_key_iv: "103e98eb7d41257693a0c276189e66be"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.970916 17168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:37.971802 17182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.972002 17168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:37.972112 17168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "4e59f694152f452f8d8a9b560c52e258"
format_stamp: "Formatted at 2026-06-20 21:59:37 on dist-test-slave-gl2g"
server_key: "d1e14eab52128e4d61cd9f91cb3e65c3"
server_key_iv: "103e98eb7d41257693a0c276189e66be"
server_key_version: "encryptionkey@0"
I20260620 21:59:37.972195 17168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:37.976161 17168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:37.976475 17168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:37.976617 17168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:37.976824 17168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:37.977149 17168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:37.977203 17168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.977245 17168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:37.977272 17168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:37.983176 17168 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:36041
I20260620 21:59:37.983218 17295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:36041 every 8 connection(s)
I20260620 21:59:37.983610 17168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:37.983860 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 17168
I20260620 21:59:37.983956 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:37.984176 7110 external_mini_cluster.cc:1428] Setting key fbcb64817838a4674be7b5bbe1144fe9
I20260620 21:59:37.987228 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.984466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:42773 (local address 127.6.241.190:40657)
0620 21:59:37.984538 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.984540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.984807 (+ 267us) server_negotiation.cc:407] Connection header received
0620 21:59:37.985464 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.985465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.985471 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.985518 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.986946 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:37.986948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:37.986948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:37.986953 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:37.986970 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:37.986974 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:37.987037 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:37.987135 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 21:59:37.987153 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:59:37.987459 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.984475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40657 (local address 127.6.241.131:42773)
0620 21:59:37.984670 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:37.984718 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:37.985349 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:37.985600 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:37.985605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:37.985615 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:37.986770 (+ 1155us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:37.986770 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:37.986804 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:37.986838 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:37.987036 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:37.987042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:37.987045 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:37.987146 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 21:59:37.987214 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260620 21:59:37.987980 17296 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40657
I20260620 21:59:37.988067 17296 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:37.988189 17296 heartbeater.cc:511] Master 127.6.241.190:40657 requested a full tablet report, sending...
I20260620 21:59:37.988543 16846 ts_manager.cc:194] Registered new tserver with Master: 4e59f694152f452f8d8a9b560c52e258 (127.6.241.131:36041)
I20260620 21:59:37.996661 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:59:37.998790 16886 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:37.998201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35776 (local address 127.6.241.190:40657)
0620 21:59:37.998272 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.998275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.998413 (+ 138us) server_negotiation.cc:407] Connection header received
0620 21:59:37.998458 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.998459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.998463 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.998514 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.998693 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35776: BlockingRecv error: recv got EOF from 127.0.0.1:35776 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:59:37.998800 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 16906
I20260620 21:59:38.003916 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 17037
I20260620 21:59:38.008658 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 17168
I20260620 21:59:38.013479 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 16815
2026-06-20T21:59:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (587 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:38 dist-test-slave-gl2g krb5kdc[17306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:38 dist-test-slave-gl2g krb5kdc[17306](info): set up 2 sockets
Jun 20 21:59:38 dist-test-slave-gl2g krb5kdc[17306](info): commencing operation
krb5kdc: starting...
W20260620 21:59:39.120318 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:59:39 dist-test-slave-gl2g krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 21:59:39.263892 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:38395
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:39.359149 17319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:39.359486 17319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:39.359534 17319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:39.360849 17319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:39.360937 17319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:39.360965 17319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:39.360996 17319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:39.361042 17319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:39.362844 17319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:39.363176 17319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:39.363443 17319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:39.368629 17324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:39.368649 17325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:39.368839 17319 server_base.cc:1034] running on GCE node
W20260620 21:59:39.368657 17327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:39.369163 17319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:39.369517 17319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:39.370702 17319 hybrid_clock.cc:648] HybridClock initialized: now 1781992779370687 us; error 31 us; skew 500 ppm
Jun 20 21:59:39 dist-test-slave-gl2g krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:39.373584 17319 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 21:59:39.374104 17319 webserver.cc:460] Webserver started at http://127.6.241.190:39667/ using document root <none> and password file <none>
I20260620 21:59:39.374380 17319 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:39.374428 17319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:39.374529 17319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:39.375808 17319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "7eb297c113334020a2f861c50ad7d05c"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "aaf27dacd4b7c19efd0d7f869b14de77"
server_key_iv: "42262c9e6fcfab4c89c7d736f8dd1ee1"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.376169 17319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "7eb297c113334020a2f861c50ad7d05c"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "aaf27dacd4b7c19efd0d7f869b14de77"
server_key_iv: "42262c9e6fcfab4c89c7d736f8dd1ee1"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.377624 17319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:39.378468 17334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.378762 17319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:39.378855 17319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "7eb297c113334020a2f861c50ad7d05c"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "aaf27dacd4b7c19efd0d7f869b14de77"
server_key_iv: "42262c9e6fcfab4c89c7d736f8dd1ee1"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.378918 17319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:39.400041 17319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:39.403038 17319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:39.403216 17319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:39.407488 17319 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:38395
I20260620 21:59:39.407527 17386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:38395 every 8 connection(s)
I20260620 21:59:39.408090 17319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:39.409632 17387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:39.410521 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 17319
I20260620 21:59:39.410645 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 21:59:39.410861 7110 external_mini_cluster.cc:1428] Setting key 80d85786fe9debb4d72755acb13ef45d
I20260620 21:59:39.413137 17387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c: Bootstrap starting.
I20260620 21:59:39.413800 17387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:39.414116 17387 log.cc:824] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:39.414942 17387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c: No bootstrap required, opened a new log
I20260620 21:59:39.416843 17387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb297c113334020a2f861c50ad7d05c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38395 } }
I20260620 21:59:39.417033 17387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:39.417083 17387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb297c113334020a2f861c50ad7d05c, State: Initialized, Role: FOLLOWER
I20260620 21:59:39.417249 17387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb297c113334020a2f861c50ad7d05c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38395 } }
I20260620 21:59:39.417371 17387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:39.417419 17387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:39.417474 17387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [term 0 FOLLOWER]: Advancing to term 1
Jun 20 21:59:39 dist-test-slave-gl2g krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:39.418303 17387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb297c113334020a2f861c50ad7d05c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38395 } }
I20260620 21:59:39.418427 17387 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb297c113334020a2f861c50ad7d05c; no voters:
I20260620 21:59:39.418668 17387 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:39.418735 17392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:39.418962 17387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:39.419001 17392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [term 1 LEADER]: Becoming Leader. State: Replica: 7eb297c113334020a2f861c50ad7d05c, State: Running, Role: LEADER
I20260620 21:59:39.419206 17392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb297c113334020a2f861c50ad7d05c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38395 } }
I20260620 21:59:39.419600 17393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb297c113334020a2f861c50ad7d05c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb297c113334020a2f861c50ad7d05c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38395 } } }
I20260620 21:59:39.419683 17393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:39.419970 17394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb297c113334020a2f861c50ad7d05c. Latest consensus state: current_term: 1 leader_uuid: "7eb297c113334020a2f861c50ad7d05c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb297c113334020a2f861c50ad7d05c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38395 } } }
I20260620 21:59:39.420313 17394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:39.420568 17399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:39.421152 17399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:39.422945 17399 catalog_manager.cc:1347] Generated new cluster ID: 265fe9769e374815b62f1196f217e436
I20260620 21:59:39.423008 17399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:39.423364 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.411473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.6.241.190:38395)
0620 21:59:39.411657 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:39.411669 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:39.411708 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:59:39.412126 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:39.412139 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:39.412199 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:39.412341 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:39.413228 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.414386 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.415332 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.415781 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.415914 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:39.418808 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:39.418813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:39.418813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:39.418828 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:39.421141 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:39.421554 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:39.421558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:39.421558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:39.421668 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:39.421943 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:39.421944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:39.421945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:39.422275 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:39.422946 (+ 671us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:39.423027 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 21:59:39.423172 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260620 21:59:39.450176 17399 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:39.450836 17399 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:39.460266 17399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c: Generated new TSK 0
I20260620 21:59:39.460536 17399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 21:59:39.489477 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:38395
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:39.574807 17413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:39.575058 17413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:39.575102 17413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:39.576386 17413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:39.576465 17413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:39.576514 17413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 21:59:39.578065 17413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:39.578361 17413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:39.578661 17413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:39.583951 17419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:39.583945 17418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:39.583945 17421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:39.584236 17413 server_base.cc:1034] running on GCE node
I20260620 21:59:39.584549 17413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:39.584838 17413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:39.586021 17413 hybrid_clock.cc:648] HybridClock initialized: now 1781992779586010 us; error 32 us; skew 500 ppm
Jun 20 21:59:39 dist-test-slave-gl2g krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:39.588773 17413 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 21:59:39.589202 17413 webserver.cc:460] Webserver started at http://127.6.241.129:41769/ using document root <none> and password file <none>
I20260620 21:59:39.589421 17413 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:39.589465 17413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:39.589560 17413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:39.590688 17413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "d66693b3a81f46aa80d60ed6a5ce5d64"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "96df5d39be7d93b6d6a07d531983e438"
server_key_iv: "1da4d0de6f9a4abc473baf0d8f377b23"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.591032 17413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "d66693b3a81f46aa80d60ed6a5ce5d64"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "96df5d39be7d93b6d6a07d531983e438"
server_key_iv: "1da4d0de6f9a4abc473baf0d8f377b23"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.592396 17413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:39.593175 17428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.593392 17413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:59:39.593480 17413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "d66693b3a81f46aa80d60ed6a5ce5d64"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "96df5d39be7d93b6d6a07d531983e438"
server_key_iv: "1da4d0de6f9a4abc473baf0d8f377b23"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.593540 17413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:39.613912 17413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:39.616706 17413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:39.616840 17413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:39.617070 17413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:39.617439 17413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:39.617478 17413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.617502 17413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:39.617554 17413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.623832 17413 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:34577
I20260620 21:59:39.623903 17541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:34577 every 8 connection(s)
I20260620 21:59:39.624275 17413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:39.625257 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 17413
I20260620 21:59:39.625355 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:39.625586 7110 external_mini_cluster.cc:1428] Setting key bcf577139457b99cfc8a577933a9ce12
Jun 20 21:59:39 dist-test-slave-gl2g krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
I20260620 21:59:39.635968 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.625155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:56221 (local address 127.6.241.190:38395)
0620 21:59:39.625200 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:39.625203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:39.625480 (+ 277us) server_negotiation.cc:407] Connection header received
0620 21:59:39.626019 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:39.626020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:39.626032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:39.626103 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:39.627205 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.628015 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.629767 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.629982 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.630040 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:39.632775 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:39.632776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:39.632776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:39.632788 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:39.634314 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:39.634865 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:39.634874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:39.634875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:39.634939 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:39.635211 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:39.635212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:39.635213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:39.635337 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:39.635461 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:39.635767 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 21:59:39.635862 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 21:59:39.636157 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.625166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:38395 (local address 127.6.241.129:56221)
0620 21:59:39.625326 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:39.625364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:39.625903 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:39.626161 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:39.626165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:39.626895 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:39.627097 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:39.627102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.628254 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.628256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:39.629663 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:39.629667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.630088 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.630089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:39.630261 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:39.630908 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:39.630919 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:39.632656 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:39.634431 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:39.634440 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:39.634442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:39.634782 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:39.635010 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:39.635011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:39.635011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:39.635137 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:39.635454 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:39.635461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:39.635619 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 21:59:39.635730 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 21:59:39.635904 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260620 21:59:39.636718 17542 heartbeater.cc:347] Connected to a master server at 127.6.241.190:38395
I20260620 21:59:39.636837 17542 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:39.637099 17542 heartbeater.cc:511] Master 127.6.241.190:38395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
I20260620 21:59:39.637737 17351 ts_manager.cc:194] Registered new tserver with Master: d66693b3a81f46aa80d60ed6a5ce5d64 (127.6.241.129:34577)
I20260620 21:59:39.638388 17351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:56221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 21:59:39.653894 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:38395
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:39.740084 17547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:39.740320 17547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:39.740345 17547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:39.741591 17547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:39.741641 17547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:39.741670 17547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 21:59:39.743216 17547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:39.743494 17547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:39.743764 17547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:39.749183 17547 server_base.cc:1034] running on GCE node
W20260620 21:59:39.749136 17555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:39.749179 17553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:39.749148 17552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:39.749555 17547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:39.749864 17547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:39.751098 17547 hybrid_clock.cc:648] HybridClock initialized: now 1781992779751082 us; error 45 us; skew 500 ppm
Jun 20 21:59:39 dist-test-slave-gl2g krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:39.753907 17547 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 21:59:39.754408 17547 webserver.cc:460] Webserver started at http://127.6.241.130:35663/ using document root <none> and password file <none>
I20260620 21:59:39.754720 17547 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:39.754794 17547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:39.754935 17547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:39.756057 17547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "dd7613643d494156976d005b249c9dc7"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "fae43f9d1e674b002d15239f7d066a5b"
server_key_iv: "df28eac609e1bd244452e67a5eaf4693"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.756461 17547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "dd7613643d494156976d005b249c9dc7"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "fae43f9d1e674b002d15239f7d066a5b"
server_key_iv: "df28eac609e1bd244452e67a5eaf4693"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.757858 17547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 21:59:39.758703 17562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.758863 17547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:59:39.758966 17547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "dd7613643d494156976d005b249c9dc7"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "fae43f9d1e674b002d15239f7d066a5b"
server_key_iv: "df28eac609e1bd244452e67a5eaf4693"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.759059 17547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:39.784986 17547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:39.787253 17547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:39.787426 17547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:39.787690 17547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:39.788053 17547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:39.788120 17547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.788161 17547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:39.788190 17547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.794147 17547 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:43103
I20260620 21:59:39.794246 17675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:43103 every 8 connection(s)
I20260620 21:59:39.794639 17547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:39.799572 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 17547
I20260620 21:59:39.799661 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:39.799852 7110 external_mini_cluster.cc:1428] Setting key d0ce15b7344d612a073f09b5572c4071
Jun 20 21:59:39 dist-test-slave-gl2g krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:39.805275 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.795491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:41461 (local address 127.6.241.190:38395)
0620 21:59:39.795566 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:39.795568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:39.795779 (+ 211us) server_negotiation.cc:407] Connection header received
0620 21:59:39.796384 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:39.796385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:39.796399 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:39.796450 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:39.797468 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.798005 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.799200 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.799419 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.799506 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:39.802147 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:39.802148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:39.802148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:39.802155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:39.803661 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:39.804190 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:39.804191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:39.804191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:39.804248 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:39.804570 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:39.804571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:39.804571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:39.804695 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:39.804774 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:39.805087 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 21:59:39.805180 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:59:39.805510 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.795474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:38395 (local address 127.6.241.130:41461)
0620 21:59:39.795649 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:39.795688 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:39.796225 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:39.796533 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:39.796536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:39.797140 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:39.797342 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:39.797346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.798112 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.798113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:39.799103 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:39.799106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.799575 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.799576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:39.799732 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:39.800412 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:39.800422 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:39.802046 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:39.803749 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:39.803757 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:39.803760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:39.804107 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:39.804338 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:39.804339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:39.804339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:39.804490 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:39.804766 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:39.804773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:39.804938 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 21:59:39.805055 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 21:59:39.805233 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260620 21:59:39.806056 17676 heartbeater.cc:347] Connected to a master server at 127.6.241.190:38395
I20260620 21:59:39.806172 17676 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:39.806383 17676 heartbeater.cc:511] Master 127.6.241.190:38395 requested a full tablet report, sending...
I20260620 21:59:39.806768 17351 ts_manager.cc:194] Registered new tserver with Master: dd7613643d494156976d005b249c9dc7 (127.6.241.130:43103)
I20260620 21:59:39.807183 17351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:41461
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 21:59:39.826260 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:38395
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:39.911674 17681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:39.911917 17681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:39.911944 17681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:39.913196 17681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:39.913249 17681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:39.913278 17681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 21:59:39.914983 17681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 21:59:39.915253 17681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:39.915503 17681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:39.920745 17689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:39.920809 17687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:39.921028 17686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:39.921010 17681 server_base.cc:1034] running on GCE node
I20260620 21:59:39.921247 17681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:39.921514 17681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:39.922660 17681 hybrid_clock.cc:648] HybridClock initialized: now 1781992779922648 us; error 38 us; skew 500 ppm
Jun 20 21:59:39 dist-test-slave-gl2g krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:39.925329 17681 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 21:59:39.925799 17681 webserver.cc:460] Webserver started at http://127.6.241.131:43819/ using document root <none> and password file <none>
I20260620 21:59:39.926034 17681 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:39.926110 17681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:39.926255 17681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:39.927434 17681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "147d2e75f1a2458aa2bc4073c0c50916"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "457ddc2bf37e147a33fb6135768d0d00"
server_key_iv: "b2b409ce6b520daa5f8280ad6bee4023"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.927807 17681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "147d2e75f1a2458aa2bc4073c0c50916"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "457ddc2bf37e147a33fb6135768d0d00"
server_key_iv: "b2b409ce6b520daa5f8280ad6bee4023"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.929172 17681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:59:39.929914 17696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.930127 17681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 21:59:39.930244 17681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "147d2e75f1a2458aa2bc4073c0c50916"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-gl2g"
server_key: "457ddc2bf37e147a33fb6135768d0d00"
server_key_iv: "b2b409ce6b520daa5f8280ad6bee4023"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.930335 17681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:39.970261 17681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:39.972561 17681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:39.972748 17681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:39.973067 17681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:39.973425 17681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:39.973487 17681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.973531 17681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:39.973557 17681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.979691 17681 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:45307
I20260620 21:59:39.979743 17809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:45307 every 8 connection(s)
I20260620 21:59:39.980139 17681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:39.982000 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 17681
I20260620 21:59:39.982120 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:39.982324 7110 external_mini_cluster.cc:1428] Setting key 6f57f601d9543e5019d14b1f5ca7272a
Jun 20 21:59:39 dist-test-slave-gl2g krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 21:59:39.990048 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.980978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:45939 (local address 127.6.241.190:38395)
0620 21:59:39.981043 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:39.981046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:39.981282 (+ 236us) server_negotiation.cc:407] Connection header received
0620 21:59:39.981814 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:39.981815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:39.981828 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:39.981869 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:39.983241 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.983766 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.984784 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.984974 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.985024 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:39.987254 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:39.987255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:39.987256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:39.987262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:39.988739 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:39.989149 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:39.989150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:39.989150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:39.989199 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:39.989434 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:39.989435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:39.989435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:39.989537 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:39.989586 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:39.989917 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 21:59:39.989980 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:59:39.990227 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.980965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:38395 (local address 127.6.241.131:45939)
0620 21:59:39.981134 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:39.981179 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:39.981686 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:39.981966 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:39.981969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:39.982849 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:39.983144 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:39.983152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.983888 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.983889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:39.984705 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:39.984708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.985046 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.985047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:39.985151 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:39.985640 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:39.985651 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:39.987169 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:39.988836 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:39.988843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:39.988845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:39.989076 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:39.989262 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:39.989263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:39.989263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:39.989378 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:39.989592 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:39.989596 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:39.989768 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 21:59:39.989882 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 21:59:39.990011 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260620 21:59:39.990793 17810 heartbeater.cc:347] Connected to a master server at 127.6.241.190:38395
I20260620 21:59:39.990906 17810 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:39.991087 17810 heartbeater.cc:511] Master 127.6.241.190:38395 requested a full tablet report, sending...
I20260620 21:59:39.991473 17351 ts_manager.cc:194] Registered new tserver with Master: 147d2e75f1a2458aa2bc4073c0c50916 (127.6.241.131:45307)
I20260620 21:59:39.991858 17351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:45939
I20260620 21:59:39.994088 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:40.001920 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.995456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.6.241.190:38395)
0620 21:59:39.995523 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:39.995526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:39.995616 (+ 90us) server_negotiation.cc:407] Connection header received
0620 21:59:39.995715 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:39.995716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:39.995726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:39.995775 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:39.996424 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.996920 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.997663 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:39.997876 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:39.997966 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:39.999288 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:39.999290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:39.999290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:39.999299 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:40.000699 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:40.001064 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:40.001065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:40.001066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:40.001126 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:40.001331 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:40.001332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:40.001333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:40.001501 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:40.001614 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:40.001717 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 21:59:40.001802 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:59:40.003509 17351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:40.005082 17351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:40.012202 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.007551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:43103 (local address 127.0.0.1:37742)
0620 21:59:40.008005 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:40.008013 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:40.008058 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:40.008571 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:40.008572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:40.008574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:40.008633 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.008635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.009523 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.009524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.010923 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.010925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.011862 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.011863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.011948 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.011965 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:59:40.012046 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 21:59:40.012105 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260620 21:59:40.012215 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.007835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.6.241.130:43103)
0620 21:59:40.008403 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:40.008408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:40.008418 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:40.008459 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:40.008461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:40.008476 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:40.008572 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:40.008718 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.009418 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.011004 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.011765 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.011833 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.011895 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:40.012080 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 21:59:40.012121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":115,"threads_started":1}
I20260620 21:59:40.013057 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.008072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.6.241.129:34577)
0620 21:59:40.008282 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:40.008286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:40.008296 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:40.008334 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:40.008336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:40.008350 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:40.008437 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:40.008637 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.009702 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.011859 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.012608 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.012676 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.012735 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:40.012925 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 21:59:40.012970 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260620 21:59:40.013070 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.007543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:45307 (local address 127.0.0.1:36884)
0620 21:59:40.007661 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:40.007706 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:40.007776 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:40.008260 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:40.008264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:40.008267 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:40.008352 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.008355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.009820 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.009821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.010911 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.010914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.012785 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.012786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.012853 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.012857 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:40.012916 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:59:40.012968 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260620 21:59:40.013072 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.007551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:34577 (local address 127.0.0.1:52498)
0620 21:59:40.008156 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:40.008165 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:40.008200 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:40.008513 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:40.008514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:40.008516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:40.008562 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.008563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.010669 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.010670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.011728 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.011732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.012770 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.012771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.012840 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.012844 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:40.012903 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:59:40.012951 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":24,"thread_start_us":94,"threads_started":1}
I20260620 21:59:40.013394 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.007855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.6.241.131:45307)
0620 21:59:40.008030 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:40.008036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:40.008048 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:59:40.008117 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:40.008119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:40.008139 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:40.008435 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:40.008509 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.009683 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.011926 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.012721 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.012787 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.012841 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:40.013261 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 21:59:40.013318 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260620 21:59:40.013473 17610 tablet_service.cc:1467] Processing CreateTablet for tablet cacee79547714c3c9f0a51270699e7bc (DEFAULT_TABLE table=test-table [id=cdfef625146446a5bae36e63fe34b646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:40.013919 17610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacee79547714c3c9f0a51270699e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:40.014262 17476 tablet_service.cc:1467] Processing CreateTablet for tablet cacee79547714c3c9f0a51270699e7bc (DEFAULT_TABLE table=test-table [id=cdfef625146446a5bae36e63fe34b646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:40.014662 17476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacee79547714c3c9f0a51270699e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:40.015257 17744 tablet_service.cc:1467] Processing CreateTablet for tablet cacee79547714c3c9f0a51270699e7bc (DEFAULT_TABLE table=test-table [id=cdfef625146446a5bae36e63fe34b646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:40.015616 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacee79547714c3c9f0a51270699e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:40.017596 17827 tablet_bootstrap.cc:492] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: Bootstrap starting.
I20260620 21:59:40.017969 17828 tablet_bootstrap.cc:492] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64: Bootstrap starting.
I20260620 21:59:40.018232 17829 tablet_bootstrap.cc:492] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: Bootstrap starting.
I20260620 21:59:40.018296 17827 tablet_bootstrap.cc:654] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:40.018647 17828 tablet_bootstrap.cc:654] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:40.019021 17829 tablet_bootstrap.cc:654] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:40.019021 17827 log.cc:824] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:40.019340 17828 log.cc:824] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:40.019619 17829 log.cc:824] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:40.019826 17827 tablet_bootstrap.cc:492] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: No bootstrap required, opened a new log
I20260620 21:59:40.019928 17827 ts_tablet_manager.cc:1397] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:40.020117 17828 tablet_bootstrap.cc:492] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64: No bootstrap required, opened a new log
I20260620 21:59:40.020183 17828 ts_tablet_manager.cc:1397] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 21:59:40.020364 17829 tablet_bootstrap.cc:492] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: No bootstrap required, opened a new log
I20260620 21:59:40.020448 17829 ts_tablet_manager.cc:1397] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:40.021837 17829 raft_consensus.cc:348] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } }
I20260620 21:59:40.022006 17829 raft_consensus.cc:374] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:40.022078 17829 raft_consensus.cc:729] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147d2e75f1a2458aa2bc4073c0c50916, State: Initialized, Role: FOLLOWER
I20260620 21:59:40.021986 17827 raft_consensus.cc:348] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } }
I20260620 21:59:40.022194 17827 raft_consensus.cc:374] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:40.022121 17828 raft_consensus.cc:348] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } }
I20260620 21:59:40.022208 17829 consensus_queue.cc:260] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } }
I20260620 21:59:40.022264 17827 raft_consensus.cc:729] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7613643d494156976d005b249c9dc7, State: Initialized, Role: FOLLOWER
I20260620 21:59:40.022292 17828 raft_consensus.cc:374] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:40.022331 17828 raft_consensus.cc:729] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66693b3a81f46aa80d60ed6a5ce5d64, State: Initialized, Role: FOLLOWER
I20260620 21:59:40.022410 17827 consensus_queue.cc:260] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } }
I20260620 21:59:40.022419 17828 consensus_queue.cc:260] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } }
I20260620 21:59:40.022526 17829 ts_tablet_manager.cc:1428] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:40.022563 17810 heartbeater.cc:503] Master 127.6.241.190:38395 was elected leader, sending a full tablet report...
I20260620 21:59:40.022742 17542 heartbeater.cc:503] Master 127.6.241.190:38395 was elected leader, sending a full tablet report...
I20260620 21:59:40.022773 17827 ts_tablet_manager.cc:1428] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:59:40.022773 17828 ts_tablet_manager.cc:1428] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 21:59:40.022993 17676 heartbeater.cc:503] Master 127.6.241.190:38395 was elected leader, sending a full tablet report...
W20260620 21:59:40.045248 17677 tablet.cc:2369] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:40.120002 17835 raft_consensus.cc:482] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:40.120173 17835 raft_consensus.cc:504] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } }
I20260620 21:59:40.120694 17835 leader_election.cc:288] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 147d2e75f1a2458aa2bc4073c0c50916 (127.6.241.131:45307), d66693b3a81f46aa80d60ed6a5ce5d64 (127.6.241.129:34577)
W20260620 21:59:40.125092 17543 tablet.cc:2369] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:40.125242 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.120974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:45307 (local address 127.6.241.130:41037)
0620 21:59:40.121588 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:40.121599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:40.121655 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:40.121809 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:40.121809 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:40.121811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:40.121862 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.121863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.122776 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.122777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.124145 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.124148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.124976 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.124977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.125052 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.125055 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:59:40.125104 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 21:59:40.125148 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":296,"thread_start_us":208,"threads_started":1}
I20260620 21:59:40.125249 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.121042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:41037 (local address 127.6.241.131:45307)
0620 21:59:40.121117 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:40.121120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:40.121682 (+ 562us) server_negotiation.cc:407] Connection header received
0620 21:59:40.121712 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:40.121713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:40.121724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:40.121793 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:40.121932 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.122661 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.124249 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.124851 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.124906 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.124917 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:40.125129 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 21:59:40.125166 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:59:40.125885 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.120967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:34577 (local address 127.6.241.130:50377)
0620 21:59:40.121054 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:40.121067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:40.121117 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:40.121294 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:40.121296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:40.121302 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:40.121531 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.121535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.122585 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.122586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.123994 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:40.123998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.125553 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.125554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:40.125658 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.125662 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:40.125731 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:59:40.125784 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":1}
I20260620 21:59:40.125908 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.121050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:50377 (local address 127.6.241.129:34577)
0620 21:59:40.121117 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:40.121119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:40.121131 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:59:40.121187 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:40.121188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:40.121195 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:40.121240 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:40.121672 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.122464 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.124108 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.124861 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.124928 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.124936 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:40.125765 (+ 829us) server_negotiation.cc:299] Negotiation successful
0620 21:59:40.125815 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:59:40.126298 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacee79547714c3c9f0a51270699e7bc" candidate_uuid: "dd7613643d494156976d005b249c9dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" is_pre_election: true
I20260620 21:59:40.126509 17496 raft_consensus.cc:2454] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7613643d494156976d005b249c9dc7 in term 0.
I20260620 21:59:40.126794 17565 leader_election.cc:302] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d66693b3a81f46aa80d60ed6a5ce5d64, dd7613643d494156976d005b249c9dc7; no voters:
I20260620 21:59:40.126942 17835 raft_consensus.cc:2781] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:40.127046 17835 raft_consensus.cc:482] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:40.127077 17835 raft_consensus.cc:3037] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:40.127159 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacee79547714c3c9f0a51270699e7bc" candidate_uuid: "dd7613643d494156976d005b249c9dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "147d2e75f1a2458aa2bc4073c0c50916" is_pre_election: true
I20260620 21:59:40.127312 17764 raft_consensus.cc:2454] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7613643d494156976d005b249c9dc7 in term 0.
I20260620 21:59:40.128103 17835 raft_consensus.cc:504] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } }
I20260620 21:59:40.128253 17835 leader_election.cc:288] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [CANDIDATE]: Term 1 election: Requested vote from peers 147d2e75f1a2458aa2bc4073c0c50916 (127.6.241.131:45307), d66693b3a81f46aa80d60ed6a5ce5d64 (127.6.241.129:34577)
I20260620 21:59:40.128508 17496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacee79547714c3c9f0a51270699e7bc" candidate_uuid: "dd7613643d494156976d005b249c9dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64"
I20260620 21:59:40.128595 17496 raft_consensus.cc:3037] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:40.129257 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacee79547714c3c9f0a51270699e7bc" candidate_uuid: "dd7613643d494156976d005b249c9dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "147d2e75f1a2458aa2bc4073c0c50916"
I20260620 21:59:40.129348 17764 raft_consensus.cc:3037] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:40.129634 17496 raft_consensus.cc:2454] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7613643d494156976d005b249c9dc7 in term 1.
I20260620 21:59:40.129813 17565 leader_election.cc:302] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d66693b3a81f46aa80d60ed6a5ce5d64, dd7613643d494156976d005b249c9dc7; no voters:
I20260620 21:59:40.129936 17835 raft_consensus.cc:2781] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:40.130151 17835 raft_consensus.cc:686] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 1 LEADER]: Becoming Leader. State: Replica: dd7613643d494156976d005b249c9dc7, State: Running, Role: LEADER
I20260620 21:59:40.130261 17835 consensus_queue.cc:237] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } }
I20260620 21:59:40.130385 17764 raft_consensus.cc:2454] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7613643d494156976d005b249c9dc7 in term 1.
I20260620 21:59:40.132072 17349 catalog_manager.cc:5495] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7613643d494156976d005b249c9dc7 (127.6.241.130). New cstate: current_term: 1 leader_uuid: "dd7613643d494156976d005b249c9dc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7613643d494156976d005b249c9dc7" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 } health_report { overall_health: UNKNOWN } } }
I20260620 21:59:40.158201 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.155824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.6.241.130:43103)
0620 21:59:40.155888 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:40.155889 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:40.155897 (+ 8us) server_negotiation.cc:407] Connection header received
0620 21:59:40.155940 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:40.155940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:40.155960 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:40.156026 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:40.156162 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.156667 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.157445 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:40.157630 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:40.157681 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:40.157830 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:40.157940 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:40.158003 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:40.158036 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 21:59:40.158056 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:59:40.160332 17496 raft_consensus.cc:1261] T cacee79547714c3c9f0a51270699e7bc P d66693b3a81f46aa80d60ed6a5ce5d64 [term 1 FOLLOWER]: Refusing update from remote peer dd7613643d494156976d005b249c9dc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:40.160333 17764 raft_consensus.cc:1261] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [term 1 FOLLOWER]: Refusing update from remote peer dd7613643d494156976d005b249c9dc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:40.160770 17835 consensus_queue.cc:1035] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "147d2e75f1a2458aa2bc4073c0c50916" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:40.160954 17837 consensus_queue.cc:1035] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66693b3a81f46aa80d60ed6a5ce5d64" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 34577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:40.164613 17841 mvcc.cc:205] Tried to move back new op lower bound from 7299042427533598720 to 7299042427414212608. Current Snapshot: MvccSnapshot[applied={T|T < 7299042427533598720}]
I20260620 21:59:40.164752 17842 mvcc.cc:205] Tried to move back new op lower bound from 7299042427533598720 to 7299042427414212608. Current Snapshot: MvccSnapshot[applied={T|T < 7299042427533598720 or (T in {7299042427533598720})}]
I20260620 21:59:40.165903 17349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:40.166010 17349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:40.166853 17349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eb297c113334020a2f861c50ad7d05c: Sending DeleteTablet for 3 replicas of tablet cacee79547714c3c9f0a51270699e7bc
I20260620 21:59:40.167167 17610 tablet_service.cc:1514] Processing DeleteTablet for tablet cacee79547714c3c9f0a51270699e7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:40 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:37742
I20260620 21:59:40.167194 17744 tablet_service.cc:1514] Processing DeleteTablet for tablet cacee79547714c3c9f0a51270699e7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:40 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:36884
I20260620 21:59:40.167416 17850 tablet_replica.cc:330] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: stopping tablet replica
I20260620 21:59:40.167433 17476 tablet_service.cc:1514] Processing DeleteTablet for tablet cacee79547714c3c9f0a51270699e7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:40 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:52498
I20260620 21:59:40.167518 17850 raft_consensus.cc:2229] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:40.167630 17850 raft_consensus.cc:2258] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:40.167999 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 17413
I20260620 21:59:40.168133 17850 ts_tablet_manager.cc:1905] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:40.170198 17850 ts_tablet_manager.cc:1918] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:40.170279 17850 log.cc:1196] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/cacee79547714c3c9f0a51270699e7bc
I20260620 21:59:40.170560 17850 ts_tablet_manager.cc:1939] T cacee79547714c3c9f0a51270699e7bc P dd7613643d494156976d005b249c9dc7: Deleting consensus metadata
I20260620 21:59:40.171007 17338 catalog_manager.cc:4841] TS dd7613643d494156976d005b249c9dc7 (127.6.241.130:43103): tablet cacee79547714c3c9f0a51270699e7bc (table test-table [id=cdfef625146446a5bae36e63fe34b646]) successfully deleted
I20260620 21:59:40.171798 17849 tablet_replica.cc:330] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: stopping tablet replica
I20260620 21:59:40.171936 17849 raft_consensus.cc:2229] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:40.172083 17849 raft_consensus.cc:2258] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:40.172559 17849 ts_tablet_manager.cc:1905] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:40.174511 17849 ts_tablet_manager.cc:1918] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:40.174626 17849 log.cc:1196] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/cacee79547714c3c9f0a51270699e7bc
I20260620 21:59:40.174906 17849 ts_tablet_manager.cc:1939] T cacee79547714c3c9f0a51270699e7bc P 147d2e75f1a2458aa2bc4073c0c50916: Deleting consensus metadata
I20260620 21:59:40.175609 17336 catalog_manager.cc:4841] TS 147d2e75f1a2458aa2bc4073c0c50916 (127.6.241.131:45307): tablet cacee79547714c3c9f0a51270699e7bc (table test-table [id=cdfef625146446a5bae36e63fe34b646]) successfully deleted
W20260620 21:59:40.176339 17337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:34577 (error 108)
I20260620 21:59:40.176559 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 17547
I20260620 21:59:40.181537 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:40.181386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:34577 (local address 127.0.0.1:52514)
0620 21:59:40.181443 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:40.181485 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:34577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 21:59:40.181684 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 17681
W20260620 21:59:40.181725 17337 catalog_manager.cc:4584] TS d66693b3a81f46aa80d60ed6a5ce5d64 (127.6.241.129:34577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cacee79547714c3c9f0a51270699e7bc: Network error: Client connection negotiation failed: client connection to 127.6.241.129:34577: connect: Connection refused (error 111)
I20260620 21:59:40.186589 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 17319
2026-06-20T21:59:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2173 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:40 dist-test-slave-gl2g krb5kdc[17855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:40 dist-test-slave-gl2g krb5kdc[17855](info): set up 2 sockets
Jun 20 21:59:40 dist-test-slave-gl2g krb5kdc[17855](info): commencing operation
krb5kdc: starting...
W20260620 21:59:41.231536 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 21:59:41 dist-test-slave-gl2g krb5kdc[17855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.241.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.241.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/rangeradmin_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/rangeradmin_127.6.241.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.241.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.241.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/rangerlookup_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/rangerlookup_127.6.241.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.241.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.241.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/HTTP_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/HTTP_127.6.241.148@KRBTEST.COM.keytab.
I20260620 21:59:41.410480 7110 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4dSvQ8/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/postgres -l logfile start
2026-06-20 21:59:43.966 UTC [17885] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 21:59:43.966 UTC [17885] LOG: listening on IPv4 address "127.6.241.148", port 47473
2026-06-20 21:59:43.970 UTC [17885] LOG: listening on Unix socket "/tmp/.s.PGSQL.47473"
2026-06-20 21:59:44.005 UTC [17888] LOG: database system was shut down at 2026-06-20 21:59:42 UTC
2026-06-20 21:59:44.009 UTC [17885] LOG: database system is ready to accept connections
I20260620 21:59:44.959543 7110 mini_postgres.cc:96] Postgres bound to 47473
2026-06-20 21:59:44.965 UTC [17896] FATAL: database "slave" does not exist
127.6.241.148:47473 - accepting connections
I20260620 21:59:44.966182 7110 mini_ranger.cc:161] Starting Ranger...
I20260620 21:59:44.981317 7110 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 21:59:45.469651 7110 mini_ranger.cc:87] Created ranger Postgres database
I20260620 21:59:45.469743 7110 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-admin
2026-06-20 21:59:45,745 [I] DB FLAVOR :POSTGRES
2026-06-20 21:59:45,745 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:59:45,745 [I] Checking connection..
2026-06-20 21:59:45,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 21:59:46,013 [I] Checking connection passed.
2026-06-20 21:59:46,014 [I] --------- Verifying version history table ---------
2026-06-20 21:59:46,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:59:46,278 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:59:46,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:59:46,542 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:59:46,542 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 21:59:46,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 21:59:46.779 UTC [17960] WARNING: there is no transaction in progress
2026-06-20 21:59:46,785 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 21:59:46,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:59:47,027 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 21:59:47,028 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:59:47,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 21:59:47,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 21:59:47,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gl2g.c.gcp-upstream.internal','N') ;"
2026-06-20 21:59:47,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:59:48,081 [I] Table x_portal_user does not exist in database ranger
2026-06-20 21:59:48,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:59:48,327 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 21:59:48,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:59:48,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:59:48,834 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 21:59:48,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 21:59:49.855 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:49.871 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:49.888 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:49.904 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:49.920 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:49.939 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.000 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.007 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.024 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.035 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.047 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.062 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.072 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.079 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.086 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50.093 UTC [18068] WARNING: there is no transaction in progress
2026-06-20 21:59:50,992 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 21:59:50,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 21:59:51,255 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 21:59:51,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:59:51,524 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 21:59:51,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:59:51,782 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 21:59:51,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:59:52,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:59:52,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gl2g.c.gcp-upstream.internal';"
2026-06-20 21:59:52,573 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 21:59:52,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 21:59:52,820 [I] DB_PATCHES have already been applied
I20260620 21:59:52.826849 7110 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-admin:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4dSvQ8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 21:59:52.826915 7110 mini_ranger.cc:193] Using host: 127.6.241.148
I20260620 21:59:52.829797 7110 mini_ranger.cc:227] Ranger admin URL: http://127.6.241.148:53101
Jun 20, 2026 9:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 21:59:53.950541 7110 mini_ranger.cc:160] Time spent starting Ranger: real 8.984s user 0.002s sys 0.007s
Jun 20, 2026 10:00:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 10:00:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 10:00:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 10:00:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 10:00:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 22:00:15.633219 7110 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.241.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.241.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/rangerkms_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/rangerkms_127.6.241.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.241.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.241.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.241.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/HTTP_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.241.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/HTTP_127.6.241.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 22:00:15 dist-test-slave-gl2g krb5kdc[17855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992815, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 22:00:15.721206 7110 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 22:00:15.730482 7110 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 22:00:16.458976 7110 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 22:00:16.459069 7110 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-kms
I20260620 22:00:16.459087 7110 mini_ranger_kms.cc:227] Using postgres at 127.6.241.148:47473
2026-06-20 22:00:16,631 [I] DB FLAVOR :POSTGRES
2026-06-20 22:00:16,632 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:00:16,632 [I] Checking connection
2026-06-20 22:00:16,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 22:00:16,893 [I] connection success
2026-06-20 22:00:16,893 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 22:00:16,894 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 22:00:16,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 22:00:17,122 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 22:00:17,123 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:00:17,123 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 22:00:17,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:47473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 22:00:17,362 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 22:00:17.779551 7110 mini_ranger_kms.cc:310] Created kms service
I20260620 22:00:17.966070 7110 mini_ranger_kms.cc:326] Created kudu user
I20260620 22:00:18.008250 7110 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 22:00:18 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992815, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:00:18.401154 7110 mini_ranger_kms.cc:384] Added ranger policy
I20260620 22:00:18.401266 7110 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-kms:/tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 22:00:18.401291 7110 mini_ranger_kms.cc:240] Using host: 127.6.241.148
I20260620 22:00:18.404969 7110 mini_ranger_kms.cc:276] Ranger KMS PID: 18248
I20260620 22:00:18.405068 7110 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.241.148:40993
Jun 20, 2026 10:00:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:00:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:00:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 10:00:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 10:00:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 10:00:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 22:00:20 dist-test-slave-gl2g krb5kdc[17855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.241.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:00:20.623147 7110 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.902s user 0.000s sys 0.155s
I20260620 22:00:20.623252 7110 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 22:00:20.623338 7110 mini_ranger_kms.cc:401] 127.6.241.148:40993/kms/v1/keys
Jun 20 22:00:20 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.241.148@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
Jun 20 22:00:21 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.241.148@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
Jun 20 22:00:23 dist-test-slave-gl2g krb5kdc[17855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:00:23.591449 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34475
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:34475
--ranger_config_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:40993/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:00:23.672313 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:00:23.672535 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:00:23.672572 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:00:23.673676 18299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:00:23.673738 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:00:23.673761 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:00:23.673789 18299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:00:23.673820 18299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:00:23.675359 18299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.241.148:40993/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:34475
--ranger_config_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:00:23.675637 18299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:00:23.675887 18299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:00:23.680454 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:00:23.680469 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:00:23.680732 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:00:23.681118 18299 server_base.cc:1034] running on GCE node
I20260620 22:00:23.681351 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:00:23.681667 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:00:23.682817 18299 hybrid_clock.cc:648] HybridClock initialized: now 1781992823682795 us; error 39 us; skew 500 ppm
Jun 20 22:00:23 dist-test-slave-gl2g krb5kdc[17855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992823, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:00:23.685216 18299 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:00:23.685683 18299 webserver.cc:460] Webserver started at http://127.6.241.190:44783/ using document root <none> and password file <none>
I20260620 22:00:23.685897 18299 fs_manager.cc:359] Metadata directory not provided
I20260620 22:00:23.685956 18299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:00:23.686100 18299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:00:23 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992823, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:00:23.738741 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "9890e907973a4f71ab6f9743526b18db"
format_stamp: "Formatted at 2026-06-20 22:00:23 on dist-test-slave-gl2g"
server_key: "fde5791fc825cb03444e1e400a86940b"
server_key_iv: "8169091b036a52ba95f42d1602b4deaf"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:23.739149 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "9890e907973a4f71ab6f9743526b18db"
format_stamp: "Formatted at 2026-06-20 22:00:23 on dist-test-slave-gl2g"
server_key: "fde5791fc825cb03444e1e400a86940b"
server_key_iv: "8169091b036a52ba95f42d1602b4deaf"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:23.740548 18299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:00:23.754292 18314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:23.754498 18299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:00:23.754576 18299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "9890e907973a4f71ab6f9743526b18db"
format_stamp: "Formatted at 2026-06-20 22:00:23 on dist-test-slave-gl2g"
server_key: "fde5791fc825cb03444e1e400a86940b"
server_key_iv: "8169091b036a52ba95f42d1602b4deaf"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:23.754714 18299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:00:23.777275 18299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:00:23.779302 18299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:00:23.779465 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:00:23.784576 18299 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:34475
I20260620 22:00:23.784641 18366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:34475 every 8 connection(s)
I20260620 22:00:23.784992 18299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:00:23.785683 18367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:00:23.787058 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 18299
I20260620 22:00:23.787166 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:00:23.787377 7110 external_mini_cluster.cc:1428] Setting key d7cf5335e20fe1296e64346a20acbe21
I20260620 22:00:23.788575 18367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db: Bootstrap starting.
I20260620 22:00:23.789296 18367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db: Neither blocks nor log segments found. Creating new log.
I20260620 22:00:23.789624 18367 log.cc:824] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db: Log is configured to *not* fsync() on all Append() calls
I20260620 22:00:23.790271 18367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db: No bootstrap required, opened a new log
I20260620 22:00:23.792784 18367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890e907973a4f71ab6f9743526b18db" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34475 } }
I20260620 22:00:23.792945 18367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:00:23.793008 18367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9890e907973a4f71ab6f9743526b18db, State: Initialized, Role: FOLLOWER
I20260620 22:00:23.793182 18367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890e907973a4f71ab6f9743526b18db" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34475 } }
I20260620 22:00:23.793279 18367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:00:23.793318 18367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:00:23.793358 18367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:00:23.793946 18367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890e907973a4f71ab6f9743526b18db" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34475 } }
I20260620 22:00:23.794062 18367 leader_election.cc:302] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9890e907973a4f71ab6f9743526b18db; no voters:
I20260620 22:00:23.794229 18367 leader_election.cc:288] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:00:23.794256 18372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:00:23.794462 18372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [term 1 LEADER]: Becoming Leader. State: Replica: 9890e907973a4f71ab6f9743526b18db, State: Running, Role: LEADER
I20260620 22:00:23.794559 18367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:00:23.794584 18372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890e907973a4f71ab6f9743526b18db" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34475 } }
I20260620 22:00:23.794957 18372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9890e907973a4f71ab6f9743526b18db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890e907973a4f71ab6f9743526b18db" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34475 } } }
I20260620 22:00:23.795097 18372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [sys.catalog]: This master's current role is: LEADER
Jun 20 22:00:23 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:00:23.795285 18372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9890e907973a4f71ab6f9743526b18db. Latest consensus state: current_term: 1 leader_uuid: "9890e907973a4f71ab6f9743526b18db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890e907973a4f71ab6f9743526b18db" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34475 } } }
I20260620 22:00:23.795466 18372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db [sys.catalog]: This master's current role is: LEADER
I20260620 22:00:23.795629 18380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:00:23.796161 18380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:00:23.796439 18367 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:00:23.801082 18389 mvcc.cc:205] Tried to move back new op lower bound from 7299042606272610304 to 7299042606263205888. Current Snapshot: MvccSnapshot[applied={T|T < 7299042606272610304}]
I20260620 22:00:23.802173 18380 catalog_manager.cc:1347] Generated new cluster ID: 67fcf8f9f62540f7a5f66c40e93031ed
I20260620 22:00:23.802243 18380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:00:23.803306 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:23.787890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.6.241.190:34475)
0620 22:00:23.788071 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:23.788080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:23.788114 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:00:23.788650 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:23.788659 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:23.788722 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:23.788873 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:00:23.789654 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:23.791212 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:23.792327 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:23.792949 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:23.793330 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:23.795830 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:00:23.795834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:00:23.795834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:00:23.795845 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:00:23.800049 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:23.800564 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:23.800566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:23.800567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:23.801161 (+ 594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:23.801402 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:23.801403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:23.801403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:23.801921 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:00:23.802589 (+ 668us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:23.802699 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 22:00:23.802999 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260620 22:00:23.814335 18380 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:00:23.814946 18380 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:00:23.821584 18380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db: Generated new TSK 0
I20260620 22:00:23.821735 18380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 22:00:24.492933 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:40993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34475
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:00:24.577087 18412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:00:24.577369 18412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:00:24.577409 18412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:00:24.578683 18412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:00:24.578742 18412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:00:24.578785 18412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:00:24.580226 18412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.241.148:40993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:00:24.580475 18412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:00:24.580698 18412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:00:24.586396 18420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:00:24.586393 18418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:00:24.586409 18417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:00:24.587054 18412 server_base.cc:1034] running on GCE node
I20260620 22:00:24.587253 18412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:00:24.587558 18412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:00:24.588717 18412 hybrid_clock.cc:648] HybridClock initialized: now 1781992824588704 us; error 30 us; skew 500 ppm
Jun 20 22:00:24 dist-test-slave-gl2g krb5kdc[17855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:00:24.591337 18412 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:00:24.591862 18412 webserver.cc:460] Webserver started at http://127.6.241.129:45277/ using document root <none> and password file <none>
I20260620 22:00:24.592108 18412 fs_manager.cc:359] Metadata directory not provided
I20260620 22:00:24.592177 18412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:00:24.592311 18412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:00:24 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:00:24.609257 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e"
format_stamp: "Formatted at 2026-06-20 22:00:24 on dist-test-slave-gl2g"
server_key: "fbdbabcc650e5512dcf53357a0c001b8"
server_key_iv: "5206db967c2704b8e7a543f922c34f37"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:24.610939 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e"
format_stamp: "Formatted at 2026-06-20 22:00:24 on dist-test-slave-gl2g"
server_key: "fbdbabcc650e5512dcf53357a0c001b8"
server_key_iv: "5206db967c2704b8e7a543f922c34f37"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:24.612885 18412 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:00:24.624886 18427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:24.625087 18412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:00:24.625172 18412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e"
format_stamp: "Formatted at 2026-06-20 22:00:24 on dist-test-slave-gl2g"
server_key: "fbdbabcc650e5512dcf53357a0c001b8"
server_key_iv: "5206db967c2704b8e7a543f922c34f37"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:24.625255 18412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:00:24.644722 18412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:00:24.647468 18412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:00:24.647611 18412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:00:24.647850 18412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:00:24.648197 18412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:00:24.648249 18412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:24.648314 18412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:00:24.648358 18412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:24.655516 18412 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:45459
I20260620 22:00:24.655638 18542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:45459 every 8 connection(s)
I20260620 22:00:24.655934 18412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:00:24 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:00:24.664389 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 18412
I20260620 22:00:24.664492 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:00:24.664712 7110 external_mini_cluster.cc:1428] Setting key d1f181e64f247f38f6df197d8aea2b92
I20260620 22:00:24.666800 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:24.656609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:34475 (local address 127.6.241.129:59525)
0620 22:00:24.656921 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 22:00:24.656977 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 22:00:24.657490 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:00:24.657765 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:00:24.657769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:00:24.657950 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:00:24.658132 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:24.658137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:24.658982 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:24.658983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:24.660303 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:24.660306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:24.660839 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:24.660840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:24.660988 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:24.661537 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:00:24.661546 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:00:24.663214 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:00:24.665266 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:00:24.665271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:00:24.665274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:00:24.665539 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:00:24.665742 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:00:24.665742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:00:24.665743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:00:24.665854 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:00:24.666129 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:00:24.666134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:00:24.666278 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 22:00:24.666388 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 22:00:24.666532 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260620 22:00:24.667317 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:24.657002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:59525 (local address 127.6.241.190:34475)
0620 22:00:24.657157 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:24.657159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:24.657169 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:00:24.657605 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:24.657606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:24.657620 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:24.657695 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:00:24.658247 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:24.658888 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:24.660406 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:24.660716 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:24.660779 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:24.663336 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:00:24.663338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:00:24.663338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:00:24.663345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:00:24.665174 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:24.665620 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:24.665621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:24.665622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:24.665679 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:24.665929 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:24.665930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:24.665930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:24.666051 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:00:24.666107 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:24.667140 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0620 22:00:24.667221 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260620 22:00:24.667995 18543 heartbeater.cc:347] Connected to a master server at 127.6.241.190:34475
I20260620 22:00:24.668102 18543 heartbeater.cc:464] Registering TS with master...
I20260620 22:00:24.668288 18543 heartbeater.cc:511] Master 127.6.241.190:34475 requested a full tablet report, sending...
I20260620 22:00:24.668823 18331 ts_manager.cc:194] Registered new tserver with Master: 484b0cfd6ede40e6a2f73c0f47ee0f7e (127.6.241.129:45459)
I20260620 22:00:24.669679 18331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:59525
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 22:00:24.691589 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:40993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34475
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:00:24.800922 18549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:00:24.801182 18549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:00:24.801218 18549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:00:24.802855 18549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:00:24.802958 18549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:00:24.803040 18549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:00:24.805146 18549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.241.148:40993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:00:24.805387 18549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:00:24.805706 18549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 22:00:24 dist-test-slave-gl2g krb5kdc[17855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 22:00:24.811784 18559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:00:24.811852 18556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:00:24.812158 18549 server_base.cc:1034] running on GCE node
W20260620 22:00:24.812515 18557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:00:24.812731 18549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:00:24.812983 18549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:00:24.814167 18549 hybrid_clock.cc:648] HybridClock initialized: now 1781992824814144 us; error 35 us; skew 500 ppm
Jun 20 22:00:24 dist-test-slave-gl2g krb5kdc[17855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:00:24.817409 18549 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:00:24.817898 18549 webserver.cc:460] Webserver started at http://127.6.241.130:42189/ using document root <none> and password file <none>
I20260620 22:00:24.818140 18549 fs_manager.cc:359] Metadata directory not provided
I20260620 22:00:24.818192 18549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:00:24.818329 18549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:00:24 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:00:24.832015 18549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "37e716e1271f4d76963bac6b52da0664"
format_stamp: "Formatted at 2026-06-20 22:00:24 on dist-test-slave-gl2g"
server_key: "81a7d8542bb1699091c9c1b984fac5a6"
server_key_iv: "efe76760cd8c4767d545dcaef48d3b96"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:24.832506 18549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "37e716e1271f4d76963bac6b52da0664"
format_stamp: "Formatted at 2026-06-20 22:00:24 on dist-test-slave-gl2g"
server_key: "81a7d8542bb1699091c9c1b984fac5a6"
server_key_iv: "efe76760cd8c4767d545dcaef48d3b96"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:24.840163 18549 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260620 22:00:24.857633 18566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:24.857860 18549 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 22:00:24.857945 18549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "37e716e1271f4d76963bac6b52da0664"
format_stamp: "Formatted at 2026-06-20 22:00:24 on dist-test-slave-gl2g"
server_key: "81a7d8542bb1699091c9c1b984fac5a6"
server_key_iv: "efe76760cd8c4767d545dcaef48d3b96"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:24.858026 18549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:00:24.869985 18549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:00:24.872224 18549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:00:24.872366 18549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:00:24.872563 18549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:00:24.872933 18549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:00:24.872980 18549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:24.873009 18549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:00:24.873065 18549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:24.880069 18549 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:43163
I20260620 22:00:24.880375 18679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:43163 every 8 connection(s)
I20260620 22:00:24.880539 18549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:00:24.887991 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 18549
I20260620 22:00:24.888119 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:00:24.888306 7110 external_mini_cluster.cc:1428] Setting key ab8df27e019b43babbe3eb93aed0ef8c
Jun 20 22:00:24 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:00:24.897439 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:24.881291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:39177 (local address 127.6.241.190:34475)
0620 22:00:24.881344 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:24.881347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:24.886562 (+ 5215us) server_negotiation.cc:407] Connection header received
0620 22:00:24.887237 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:24.887239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:24.887255 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:24.887311 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:00:24.888531 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:24.889236 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:24.890757 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:24.891066 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:24.891107 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:24.893696 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:00:24.893697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:00:24.893698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:00:24.893705 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:00:24.895632 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:24.896294 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:24.896295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:24.896295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:24.896364 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:24.896651 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:24.896652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:24.896652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:24.896790 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:00:24.896836 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:24.897223 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 22:00:24.897319 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:00:24.897933 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:24.881271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:34475 (local address 127.6.241.130:39177)
0620 22:00:24.886414 (+ 5143us) negotiation.cc:107] Waiting for socket to connect
0620 22:00:24.886464 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 22:00:24.887070 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:00:24.887400 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:00:24.887404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:00:24.887581 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:00:24.887760 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:24.887764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:24.889329 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:24.889330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:24.890660 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:24.890663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:24.891210 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:24.891211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:24.891356 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:24.891909 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:00:24.891918 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:00:24.893589 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:00:24.895762 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:00:24.895768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:00:24.895770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:00:24.896203 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:00:24.896441 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:00:24.896442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:00:24.896442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:00:24.896570 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:00:24.896885 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:00:24.896891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:00:24.897056 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 22:00:24.897585 (+ 529us) client_negotiation.cc:240] Negotiation successful
0620 22:00:24.897726 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5079,"thread_start_us":195,"threads_started":1}
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
I20260620 22:00:24.898478 18680 heartbeater.cc:347] Connected to a master server at 127.6.241.190:34475
I20260620 22:00:24.898568 18680 heartbeater.cc:464] Registering TS with master...
I20260620 22:00:24.898769 18680 heartbeater.cc:511] Master 127.6.241.190:34475 requested a full tablet report, sending...
I20260620 22:00:24.899185 18331 ts_manager.cc:194] Registered new tserver with Master: 37e716e1271f4d76963bac6b52da0664 (127.6.241.130:43163)
I20260620 22:00:24.899567 18331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:39177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 22:00:24.915486 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:40993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34475
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:00:25.002897 18686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:00:25.003108 18686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:00:25.003157 18686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:00:25.004297 18686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:00:25.004364 18686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:00:25.004410 18686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:00:25.005975 18686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.241.148:40993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:00:25.006219 18686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:00:25.006474 18686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:00:25.011490 18692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:00:25.011567 18694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:00:25.011607 18686 server_base.cc:1034] running on GCE node
W20260620 22:00:25.011734 18691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:00:25.011919 18686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:00:25.012161 18686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:00:25.013320 18686 hybrid_clock.cc:648] HybridClock initialized: now 1781992825013296 us; error 42 us; skew 500 ppm
Jun 20 22:00:25 dist-test-slave-gl2g krb5kdc[17855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:00:25.016032 18686 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:00:25.016572 18686 webserver.cc:460] Webserver started at http://127.6.241.131:41075/ using document root <none> and password file <none>
I20260620 22:00:25.016844 18686 fs_manager.cc:359] Metadata directory not provided
I20260620 22:00:25.016918 18686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:00:25.017051 18686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:00:25 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:00:25.033120 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "0f75258f8c1d4e38a98a5927be93a76b"
format_stamp: "Formatted at 2026-06-20 22:00:25 on dist-test-slave-gl2g"
server_key: "c720fd4491b5031d0fde30dbed3d4210"
server_key_iv: "5ea4f1ab8b5c17da1cc9ba556da78f88"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:25.033499 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "0f75258f8c1d4e38a98a5927be93a76b"
format_stamp: "Formatted at 2026-06-20 22:00:25 on dist-test-slave-gl2g"
server_key: "c720fd4491b5031d0fde30dbed3d4210"
server_key_iv: "5ea4f1ab8b5c17da1cc9ba556da78f88"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:25.038441 18686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 22:00:25.070668 18701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:25.070878 18686 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260620 22:00:25.070981 18686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "0f75258f8c1d4e38a98a5927be93a76b"
format_stamp: "Formatted at 2026-06-20 22:00:25 on dist-test-slave-gl2g"
server_key: "c720fd4491b5031d0fde30dbed3d4210"
server_key_iv: "5ea4f1ab8b5c17da1cc9ba556da78f88"
server_key_version: "kuduclusterkey@0"
I20260620 22:00:25.071089 18686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:00:25.091130 18686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:00:25.093611 18686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:00:25.093832 18686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:00:25.094127 18686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:00:25.094574 18686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:00:25.094687 18686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:25.094767 18686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:00:25.094823 18686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:00:25.105404 18686 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:46589
I20260620 22:00:25.105855 18686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:00:25.106226 18814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:46589 every 8 connection(s)
I20260620 22:00:25.110759 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 18686
I20260620 22:00:25.110872 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:00:25.111075 7110 external_mini_cluster.cc:1428] Setting key ed0ad76ebb9f293725f41af1c717683a
Jun 20 22:00:25 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:00:25.118301 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.107749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:60993 (local address 127.6.241.190:34475)
0620 22:00:25.107795 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:25.107797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:25.108080 (+ 283us) server_negotiation.cc:407] Connection header received
0620 22:00:25.108766 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:25.108767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:25.108779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:25.108810 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:00:25.109594 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.110348 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.112106 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.112409 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.112447 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.115006 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:00:25.115008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:00:25.115008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:00:25.115015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:00:25.116748 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:25.117229 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:25.117230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:25.117230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:25.117290 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:25.117583 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:25.117584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:25.117584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:25.117717 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:00:25.117789 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:25.118124 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 22:00:25.118207 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:00:25.118461 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.107701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:34475 (local address 127.6.241.131:60993)
0620 22:00:25.107959 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 22:00:25.107998 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:00:25.108645 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:00:25.108999 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:00:25.109004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:00:25.109229 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:00:25.109482 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.109489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.110462 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.110462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.112002 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.112005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.112561 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.112562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.112693 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.113262 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:00:25.113273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:00:25.114912 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:00:25.116839 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:00:25.116845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:00:25.116847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:00:25.117147 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:00:25.117368 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:00:25.117369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:00:25.117369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:00:25.117505 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:00:25.117798 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:00:25.117804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:00:25.117970 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 22:00:25.118091 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 22:00:25.118237 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260620 22:00:25.119016 18815 heartbeater.cc:347] Connected to a master server at 127.6.241.190:34475
I20260620 22:00:25.119115 18815 heartbeater.cc:464] Registering TS with master...
I20260620 22:00:25.119288 18815 heartbeater.cc:511] Master 127.6.241.190:34475 requested a full tablet report, sending...
I20260620 22:00:25.119645 18331 ts_manager.cc:194] Registered new tserver with Master: 0f75258f8c1d4e38a98a5927be93a76b (127.6.241.131:46589)
I20260620 22:00:25.120129 18331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:60993
I20260620 22:00:25.122707 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:00:25.132356 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.124311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.6.241.190:34475)
0620 22:00:25.124345 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:25.124347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:25.124599 (+ 252us) server_negotiation.cc:407] Connection header received
0620 22:00:25.124687 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:25.124688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:25.124715 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:25.124759 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:00:25.125560 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.126321 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.127427 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.127715 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.127754 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.129208 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:00:25.129209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:00:25.129210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:00:25.129216 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:00:25.130890 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:25.131524 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:25.131525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:25.131525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:25.131589 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:00:25.131772 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:00:25.131773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:00:25.131773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:00:25.131894 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:00:25.131968 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:25.132198 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 22:00:25.132271 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:00:25.134384 18331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:00:25.136637 18331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:00:25.146229 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.140850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:45459 (local address 127.0.0.1:47362)
0620 22:00:25.141603 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 22:00:25.141612 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:00:25.141660 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:00:25.141945 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:00:25.141948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:00:25.141950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:00:25.142007 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.142009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.143382 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.143383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.144829 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.144832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.145928 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.145929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.146012 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.146026 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:00:25.146097 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:00:25.146146 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":87,"threads_started":1}
I20260620 22:00:25.147096 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.141210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.6.241.129:45459)
0620 22:00:25.141346 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:25.141351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:25.141681 (+ 330us) server_negotiation.cc:407] Connection header received
0620 22:00:25.141731 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:25.141734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:25.141751 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:25.141837 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:00:25.142137 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.143285 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.144941 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.145729 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.145802 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.145891 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:25.146962 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0620 22:00:25.147005 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260620 22:00:25.147799 18472 tablet_service.cc:1467] Processing CreateTablet for tablet d32a3be3e8164bb5bbd3e5e676b89689 (DEFAULT_TABLE table=test-table [id=8fe1a725ad8c493eb2510ba3bccd2d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:00:25.148170 18472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32a3be3e8164bb5bbd3e5e676b89689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:00:25.148998 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.141027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:43163 (local address 127.0.0.1:60164)
0620 22:00:25.141734 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 22:00:25.141744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:00:25.141775 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:00:25.145349 (+ 3574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:00:25.145351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:00:25.145353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:00:25.145393 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.145394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.146710 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.146711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.147794 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.147797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.148741 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.148743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.148809 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.148814 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:00:25.148877 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 22:00:25.148929 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":55,"threads_started":1}
I20260620 22:00:25.149017 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.142225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.6.241.130:43163)
0620 22:00:25.145175 (+ 2950us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:25.145180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:25.145192 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:00:25.145231 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:25.145232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:25.145247 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:25.145333 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:00:25.145488 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.146569 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.147913 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.148645 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.148720 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.148789 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:25.148898 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 22:00:25.148939 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2913,"thread_start_us":72,"threads_started":1}
I20260620 22:00:25.151291 18596 tablet_service.cc:1467] Processing CreateTablet for tablet d32a3be3e8164bb5bbd3e5e676b89689 (DEFAULT_TABLE table=test-table [id=8fe1a725ad8c493eb2510ba3bccd2d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:00:25.151644 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32a3be3e8164bb5bbd3e5e676b89689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:00:25.155990 18833 tablet_bootstrap.cc:492] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: Bootstrap starting.
I20260620 22:00:25.156059 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.140812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:46589 (local address 127.0.0.1:36272)
0620 22:00:25.141399 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 22:00:25.141437 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:00:25.141503 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:00:25.151752 (+ 10249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:00:25.151753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:00:25.151755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:00:25.151790 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.151791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.153757 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.153758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.154858 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.154860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.155820 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.155821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.155893 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.155899 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:00:25.155934 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 22:00:25.155972 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":73,"threads_started":1}
I20260620 22:00:25.156492 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.141061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.6.241.131:46589)
0620 22:00:25.151558 (+ 10497us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:25.151563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:25.151575 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:00:25.151612 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:25.151614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:25.151629 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:25.151874 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:00:25.151944 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.153653 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.154978 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.155720 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.156238 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.156327 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:25.156353 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 22:00:25.156407 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10458,"thread_start_us":66,"threads_started":1}
I20260620 22:00:25.156716 18834 tablet_bootstrap.cc:492] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e: Bootstrap starting.
I20260620 22:00:25.157104 18739 tablet_service.cc:1467] Processing CreateTablet for tablet d32a3be3e8164bb5bbd3e5e676b89689 (DEFAULT_TABLE table=test-table [id=8fe1a725ad8c493eb2510ba3bccd2d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:00:25.157413 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32a3be3e8164bb5bbd3e5e676b89689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:00:25.157635 18834 tablet_bootstrap.cc:654] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e: Neither blocks nor log segments found. Creating new log.
I20260620 22:00:25.157943 18834 log.cc:824] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:00:25.158713 18833 tablet_bootstrap.cc:654] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: Neither blocks nor log segments found. Creating new log.
I20260620 22:00:25.159049 18833 log.cc:824] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: Log is configured to *not* fsync() on all Append() calls
I20260620 22:00:25.159811 18834 tablet_bootstrap.cc:492] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e: No bootstrap required, opened a new log
I20260620 22:00:25.159888 18834 ts_tablet_manager.cc:1397] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:00:25.161855 18834 raft_consensus.cc:348] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } }
I20260620 22:00:25.162005 18834 raft_consensus.cc:374] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:00:25.162045 18834 raft_consensus.cc:729] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484b0cfd6ede40e6a2f73c0f47ee0f7e, State: Initialized, Role: FOLLOWER
I20260620 22:00:25.162178 18834 consensus_queue.cc:260] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } }
I20260620 22:00:25.162477 18834 ts_tablet_manager.cc:1428] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:00:25.162637 18543 heartbeater.cc:503] Master 127.6.241.190:34475 was elected leader, sending a full tablet report...
I20260620 22:00:25.163126 18833 tablet_bootstrap.cc:492] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: No bootstrap required, opened a new log
I20260620 22:00:25.163205 18833 ts_tablet_manager.cc:1397] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260620 22:00:25.164474 18833 raft_consensus.cc:348] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } }
I20260620 22:00:25.164615 18833 raft_consensus.cc:374] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:00:25.164655 18833 raft_consensus.cc:729] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e716e1271f4d76963bac6b52da0664, State: Initialized, Role: FOLLOWER
I20260620 22:00:25.164769 18833 consensus_queue.cc:260] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } }
I20260620 22:00:25.165023 18680 heartbeater.cc:503] Master 127.6.241.190:34475 was elected leader, sending a full tablet report...
I20260620 22:00:25.165012 18833 ts_tablet_manager.cc:1428] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:00:25.166060 18837 tablet_bootstrap.cc:492] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: Bootstrap starting.
I20260620 22:00:25.167487 18837 tablet_bootstrap.cc:654] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: Neither blocks nor log segments found. Creating new log.
I20260620 22:00:25.167742 18837 log.cc:824] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:00:25.168397 18837 tablet_bootstrap.cc:492] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: No bootstrap required, opened a new log
I20260620 22:00:25.168478 18837 ts_tablet_manager.cc:1397] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:00:25.170089 18837 raft_consensus.cc:348] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } }
I20260620 22:00:25.170233 18837 raft_consensus.cc:374] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:00:25.170274 18837 raft_consensus.cc:729] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f75258f8c1d4e38a98a5927be93a76b, State: Initialized, Role: FOLLOWER
I20260620 22:00:25.170397 18837 consensus_queue.cc:260] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } }
I20260620 22:00:25.170673 18815 heartbeater.cc:503] Master 127.6.241.190:34475 was elected leader, sending a full tablet report...
I20260620 22:00:25.170666 18837 ts_tablet_manager.cc:1428] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:00:25.218485 18838 raft_consensus.cc:482] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:00:25.218633 18838 raft_consensus.cc:504] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } }
I20260620 22:00:25.219039 18838 leader_election.cc:288] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f75258f8c1d4e38a98a5927be93a76b (127.6.241.131:46589), 37e716e1271f4d76963bac6b52da0664 (127.6.241.130:43163)
I20260620 22:00:25.223623 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.219419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:33639 (local address 127.6.241.131:46589)
0620 22:00:25.219462 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:25.219464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:25.219547 (+ 83us) server_negotiation.cc:407] Connection header received
0620 22:00:25.219649 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:25.219650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:25.219660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:25.219692 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:00:25.219932 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.220711 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.222651 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.223187 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.223239 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.223248 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:25.223519 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 22:00:25.223555 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:00:25.224063 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.219176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:46589 (local address 127.6.241.129:33639)
0620 22:00:25.219485 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 22:00:25.219495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:00:25.219573 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:00:25.219770 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:00:25.219771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:00:25.219773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:00:25.219854 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.219856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.220832 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.220833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.222520 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.222523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.223320 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.223321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.223419 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.223422 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:00:25.223939 (+ 517us) client_negotiation.cc:240] Negotiation successful
0620 22:00:25.223983 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":72,"threads_started":1}
I20260620 22:00:25.224491 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32a3be3e8164bb5bbd3e5e676b89689" candidate_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f75258f8c1d4e38a98a5927be93a76b" is_pre_election: true
I20260620 22:00:25.224649 18762 raft_consensus.cc:2454] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484b0cfd6ede40e6a2f73c0f47ee0f7e in term 0.
I20260620 22:00:25.224938 18431 leader_election.cc:302] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f75258f8c1d4e38a98a5927be93a76b, 484b0cfd6ede40e6a2f73c0f47ee0f7e; no voters:
I20260620 22:00:25.225055 18838 raft_consensus.cc:2781] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:00:25.225091 18838 raft_consensus.cc:482] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:00:25.225112 18838 raft_consensus.cc:3037] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:00:25.225943 18838 raft_consensus.cc:504] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } }
I20260620 22:00:25.226070 18838 leader_election.cc:288] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [CANDIDATE]: Term 1 election: Requested vote from peers 0f75258f8c1d4e38a98a5927be93a76b (127.6.241.131:46589), 37e716e1271f4d76963bac6b52da0664 (127.6.241.130:43163)
I20260620 22:00:25.226382 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32a3be3e8164bb5bbd3e5e676b89689" candidate_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f75258f8c1d4e38a98a5927be93a76b"
I20260620 22:00:25.226469 18762 raft_consensus.cc:3037] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:00:25.227325 18762 raft_consensus.cc:2454] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484b0cfd6ede40e6a2f73c0f47ee0f7e in term 1.
I20260620 22:00:25.227524 18431 leader_election.cc:302] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f75258f8c1d4e38a98a5927be93a76b, 484b0cfd6ede40e6a2f73c0f47ee0f7e; no voters:
I20260620 22:00:25.227638 18838 raft_consensus.cc:2781] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:00:25.227770 18838 raft_consensus.cc:686] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [term 1 LEADER]: Becoming Leader. State: Replica: 484b0cfd6ede40e6a2f73c0f47ee0f7e, State: Running, Role: LEADER
I20260620 22:00:25.227854 18838 consensus_queue.cc:237] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } }
I20260620 22:00:25.228132 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.219368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:43979 (local address 127.6.241.130:43163)
0620 22:00:25.219412 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:25.219414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:25.219789 (+ 375us) server_negotiation.cc:407] Connection header received
0620 22:00:25.219912 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:25.219913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:25.219922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:25.219950 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:00:25.225238 (+ 5288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.225885 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.227203 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.227773 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.227818 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.227827 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:25.228038 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 22:00:25.228074 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:00:25.228614 18331 catalog_manager.cc:5495] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 484b0cfd6ede40e6a2f73c0f47ee0f7e (127.6.241.129). New cstate: current_term: 1 leader_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 45459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 } health_report { overall_health: UNKNOWN } } }
I20260620 22:00:25.229784 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.219202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:43163 (local address 127.6.241.129:43979)
0620 22:00:25.219728 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 22:00:25.219738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:00:25.219812 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:00:25.220012 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:00:25.220013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:00:25.220014 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:00:25.220062 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.220063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.226000 (+ 5937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.226001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.227102 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:00:25.227104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.227893 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.227894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:00:25.227957 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.227960 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:00:25.229129 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0620 22:00:25.229173 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":69,"threads_started":1}
I20260620 22:00:25.230089 18633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32a3be3e8164bb5bbd3e5e676b89689" candidate_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e716e1271f4d76963bac6b52da0664"
I20260620 22:00:25.230085 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32a3be3e8164bb5bbd3e5e676b89689" candidate_uuid: "484b0cfd6ede40e6a2f73c0f47ee0f7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e716e1271f4d76963bac6b52da0664" is_pre_election: true
I20260620 22:00:25.230224 18633 raft_consensus.cc:3037] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:00:25.231153 18633 raft_consensus.cc:2454] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484b0cfd6ede40e6a2f73c0f47ee0f7e in term 1.
I20260620 22:00:25.249711 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.246393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.6.241.129:45459)
0620 22:00:25.246440 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:00:25.246444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:00:25.246461 (+ 17us) server_negotiation.cc:407] Connection header received
0620 22:00:25.246492 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:00:25.246493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:00:25.246511 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:00:25.246577 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:00:25.246765 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.247500 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.248613 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:00:25.248897 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:00:25.248962 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:00:25.249161 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:00:25.249300 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:00:25.249375 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:00:25.249593 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 22:00:25.249613 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 22:00:25 dist-test-slave-gl2g krb5kdc[17855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:00:25.263675 18762 raft_consensus.cc:1261] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [term 1 FOLLOWER]: Refusing update from remote peer 484b0cfd6ede40e6a2f73c0f47ee0f7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:00:25.264017 18844 consensus_queue.cc:1035] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f75258f8c1d4e38a98a5927be93a76b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:00:25.265981 18633 raft_consensus.cc:1261] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664 [term 1 FOLLOWER]: Refusing update from remote peer 484b0cfd6ede40e6a2f73c0f47ee0f7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:00:25.266628 18844 consensus_queue.cc:1035] T d32a3be3e8164bb5bbd3e5e676b89689 P 484b0cfd6ede40e6a2f73c0f47ee0f7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e716e1271f4d76963bac6b52da0664" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 43163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:00:25.271579 18847 mvcc.cc:205] Tried to move back new op lower bound from 7299042612276428800 to 7299042612133896192. Current Snapshot: MvccSnapshot[applied={T|T < 7299042612276428800}]
I20260620 22:00:25.271674 18848 mvcc.cc:205] Tried to move back new op lower bound from 7299042612276428800 to 7299042612133896192. Current Snapshot: MvccSnapshot[applied={T|T < 7299042612276428800}]
I20260620 22:00:25.273881 18850 mvcc.cc:205] Tried to move back new op lower bound from 7299042612276428800 to 7299042612133896192. Current Snapshot: MvccSnapshot[applied={T|T < 7299042612276428800}]
I20260620 22:00:25.274530 18331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:00:25.274631 18331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:00:25.275285 18331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9890e907973a4f71ab6f9743526b18db: Sending DeleteTablet for 3 replicas of tablet d32a3be3e8164bb5bbd3e5e676b89689
I20260620 22:00:25.275635 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet d32a3be3e8164bb5bbd3e5e676b89689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:00:25 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:60164
I20260620 22:00:25.275636 18472 tablet_service.cc:1514] Processing DeleteTablet for tablet d32a3be3e8164bb5bbd3e5e676b89689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:00:25 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:47362
I20260620 22:00:25.275988 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet d32a3be3e8164bb5bbd3e5e676b89689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:00:25 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:36272
I20260620 22:00:25.276535 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 18412
I20260620 22:00:25.278872 18859 tablet_replica.cc:330] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: stopping tablet replica
I20260620 22:00:25.278980 18859 raft_consensus.cc:2229] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:00:25.279096 18859 raft_consensus.cc:2258] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:00:25.279244 18858 tablet_replica.cc:330] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: stopping tablet replica
I20260620 22:00:25.279346 18858 raft_consensus.cc:2229] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:00:25.279438 18858 raft_consensus.cc:2258] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:00:25.279945 18858 ts_tablet_manager.cc:1905] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:00:25.281543 18858 ts_tablet_manager.cc:1918] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:00:25.281618 18858 log.cc:1196] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/d32a3be3e8164bb5bbd3e5e676b89689
I20260620 22:00:25.281968 18859 ts_tablet_manager.cc:1905] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:00:25.282193 18858 ts_tablet_manager.cc:1939] T d32a3be3e8164bb5bbd3e5e676b89689 P 37e716e1271f4d76963bac6b52da0664: Deleting consensus metadata
I20260620 22:00:25.282624 18315 catalog_manager.cc:4841] TS 37e716e1271f4d76963bac6b52da0664 (127.6.241.130:43163): tablet d32a3be3e8164bb5bbd3e5e676b89689 (table test-table [id=8fe1a725ad8c493eb2510ba3bccd2d37]) successfully deleted
I20260620 22:00:25.283638 18859 ts_tablet_manager.cc:1918] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:00:25.283708 18859 log.cc:1196] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/d32a3be3e8164bb5bbd3e5e676b89689
I20260620 22:00:25.284282 18859 ts_tablet_manager.cc:1939] T d32a3be3e8164bb5bbd3e5e676b89689 P 0f75258f8c1d4e38a98a5927be93a76b: Deleting consensus metadata
I20260620 22:00:25.284729 18318 catalog_manager.cc:4841] TS 0f75258f8c1d4e38a98a5927be93a76b (127.6.241.131:46589): tablet d32a3be3e8164bb5bbd3e5e676b89689 (table test-table [id=8fe1a725ad8c493eb2510ba3bccd2d37]) successfully deleted
W20260620 22:00:25.286589 18315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:45459 (error 108)
I20260620 22:00:25.286899 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 18549
I20260620 22:00:25.287853 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:00:25.287739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:45459 (local address 127.0.0.1:47384)
0620 22:00:25.287787 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:00:25.287820 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:45459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:00:25.287955 18315 catalog_manager.cc:4584] TS 484b0cfd6ede40e6a2f73c0f47ee0f7e (127.6.241.129:45459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d32a3be3e8164bb5bbd3e5e676b89689: Network error: Client connection negotiation failed: client connection to 127.6.241.129:45459: connect: Connection refused (error 111)
I20260620 22:00:25.292765 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 18686
I20260620 22:00:25.297761 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 18299
I20260620 22:00:25.303658 7110 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 22:00:25.648442 7110 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 22:00:25.648554 7110 mini_ranger.cc:66] Stopping Ranger...
I20260620 22:00:25.994066 7110 mini_ranger.cc:68] Stopped Ranger
2026-06-20 22:00:25.994 UTC [17885] LOG: received smart shutdown request
2026-06-20 22:00:25.997 UTC [17885] LOG: background worker "logical replication launcher" (PID 17894) exited with exit code 1
2026-06-20 22:00:25.997 UTC [17889] LOG: shutting down
2026-06-20 22:00:26.136 UTC [17885] LOG: database system is shut down
2026-06-20T22:00:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46059 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:00:26 dist-test-slave-gl2g krb5kdc[18869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:00:26 dist-test-slave-gl2g krb5kdc[18869](info): set up 2 sockets
Jun 20 22:00:26 dist-test-slave-gl2g krb5kdc[18869](info): commencing operation
krb5kdc: starting...
W20260620 22:00:27.278362 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:00:27 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:00:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.241.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.241.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/rangeradmin_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/rangeradmin_127.6.241.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.241.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.241.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/rangerlookup_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/rangerlookup_127.6.241.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.241.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.241.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/HTTP_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/HTTP_127.6.241.148@KRBTEST.COM.keytab.
I20260620 22:00:27.434338 7110 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4dSvQ8/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/postgres -l logfile start
2026-06-20 22:00:29.711 UTC [18899] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 22:00:29.711 UTC [18899] LOG: listening on IPv4 address "127.6.241.148", port 34361
2026-06-20 22:00:29.714 UTC [18899] LOG: listening on Unix socket "/tmp/.s.PGSQL.34361"
2026-06-20 22:00:29.735 UTC [18902] LOG: database system was shut down at 2026-06-20 22:00:28 UTC
2026-06-20 22:00:29.738 UTC [18899] LOG: database system is ready to accept connections
I20260620 22:00:30.690635 7110 mini_postgres.cc:96] Postgres bound to 34361
2026-06-20 22:00:30.696 UTC [18910] FATAL: database "slave" does not exist
127.6.241.148:34361 - accepting connections
I20260620 22:00:30.696830 7110 mini_ranger.cc:161] Starting Ranger...
I20260620 22:00:30.711184 7110 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 22:00:31.138967 7110 mini_ranger.cc:87] Created ranger Postgres database
I20260620 22:00:31.139045 7110 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-admin
2026-06-20 22:00:31,396 [I] DB FLAVOR :POSTGRES
2026-06-20 22:00:31,397 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:00:31,397 [I] Checking connection..
2026-06-20 22:00:31,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 22:00:31,632 [I] Checking connection passed.
2026-06-20 22:00:31,633 [I] --------- Verifying version history table ---------
2026-06-20 22:00:31,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:00:31,868 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:00:31,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:00:32,104 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:00:32,104 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 22:00:32,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 22:00:32.354 UTC [18974] WARNING: there is no transaction in progress
2026-06-20 22:00:32,360 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 22:00:32,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:00:32,613 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 22:00:32,613 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:00:32,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 22:00:32,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 22:00:33,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gl2g.c.gcp-upstream.internal','N') ;"
2026-06-20 22:00:33,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:00:33,555 [I] Table x_portal_user does not exist in database ranger
2026-06-20 22:00:33,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:00:33,795 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 22:00:33,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:00:34,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:00:34,299 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 22:00:34,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 22:00:35.200 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.215 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.230 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.244 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.262 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.277 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.329 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.335 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.348 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.357 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.366 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.379 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.388 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.394 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.400 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35.406 UTC [19082] WARNING: there is no transaction in progress
2026-06-20 22:00:35,963 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 22:00:35,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 22:00:36,201 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 22:00:36,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:00:36,425 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 22:00:36,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:00:36,645 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 22:00:36,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:00:36,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:00:37,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gl2g.c.gcp-upstream.internal';"
2026-06-20 22:00:37,324 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 22:00:37,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 22:00:37,569 [I] DB_PATCHES have already been applied
I20260620 22:00:37.574818 7110 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-admin:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4dSvQ8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 22:00:37.574919 7110 mini_ranger.cc:193] Using host: 127.6.241.148
I20260620 22:00:37.577880 7110 mini_ranger.cc:227] Ranger admin URL: http://127.6.241.148:46081
Jun 20, 2026 10:00:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:00:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:00:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 10:00:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 10:00:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 10:00:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 10:00:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 22:00:38.690474 7110 mini_ranger.cc:160] Time spent starting Ranger: real 7.994s user 0.001s sys 0.007s
Jun 20, 2026 10:00:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 10:00:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 10:00:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 10:00:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 10:00:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 22:00:58.336335 7110 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.241.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.241.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/rangerkms_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.241.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/rangerkms_127.6.241.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.241.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.241.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.241.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/HTTP_127.6.241.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.241.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/HTTP_127.6.241.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 22:00:58 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 22:00:58.424374 7110 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 22:00:58.433782 7110 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 22:00:59.200893 7110 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 22:00:59.201001 7110 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-kms
I20260620 22:00:59.201025 7110 mini_ranger_kms.cc:227] Using postgres at 127.6.241.148:34361
2026-06-20 22:00:59,361 [I] DB FLAVOR :POSTGRES
2026-06-20 22:00:59,361 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:00:59,362 [I] Checking connection
2026-06-20 22:00:59,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 22:00:59,580 [I] connection success
2026-06-20 22:00:59,580 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 22:00:59,581 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 22:00:59,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 22:00:59,797 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 22:00:59,797 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:00:59,797 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 22:00:59,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.241.148:34361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 22:01:00,039 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 22:01:00.501425 7110 mini_ranger_kms.cc:310] Created kms service
I20260620 22:01:00.706082 7110 mini_ranger_kms.cc:326] Created kudu user
I20260620 22:01:00.746749 7110 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 22:01:00 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:01.121243 7110 mini_ranger_kms.cc:384] Added ranger policy
I20260620 22:01:01.121447 7110 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-kms:/tmp/dist-test-task4dSvQ8/build/release/bin/postgresql.jar:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4dSvQ8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 22:01:01.121471 7110 mini_ranger_kms.cc:240] Using host: 127.6.241.148
I20260620 22:01:01.125181 7110 mini_ranger_kms.cc:276] Ranger KMS PID: 19262
I20260620 22:01:01.125250 7110 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.241.148:60133
Jun 20, 2026 10:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 10:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 10:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 10:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 22:01:03 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.241.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:03.356814 7110 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.932s user 0.000s sys 0.144s
I20260620 22:01:03.356894 7110 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 22:01:03.356930 7110 mini_ranger_kms.cc:401] 127.6.241.148:60133/kms/v1/keys
Jun 20 22:01:03 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.241.148@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
Jun 20 22:01:04 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.241.148@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
Jun 20 22:01:06 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:01:06.154714 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:43371
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:43371
--ranger_config_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:60133/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:06.244091 19313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:06.244277 19313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:06.244297 19313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:06.245388 19313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:06.245433 19313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:06.245447 19313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:06.245460 19313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:06.245471 19313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:06.247011 19313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.241.148:60133/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:43371
--ranger_config_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:06.247246 19313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:06.247478 19313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:06.252449 19321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:06.252480 19319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:06.252487 19313 server_base.cc:1034] running on GCE node
W20260620 22:01:06.252697 19318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:06.252985 19313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:06.253309 19313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:06.254495 19313 hybrid_clock.cc:648] HybridClock initialized: now 1781992866254471 us; error 39 us; skew 500 ppm
Jun 20 22:01:06 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992866, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:06.257473 19313 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:01:06.257949 19313 webserver.cc:460] Webserver started at http://127.6.241.190:35571/ using document root <none> and password file <none>
I20260620 22:01:06.258173 19313 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:06.258212 19313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:06.258334 19313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:01:06 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992866, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:06.302420 19313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "f0c4b34935c94ea099e1841394037d53"
format_stamp: "Formatted at 2026-06-20 22:01:06 on dist-test-slave-gl2g"
server_key: "00d640bb65bbba564fd0dc5916f4d34d"
server_key_iv: "861bf55a74ccf2a6eafdf38ace14fed0"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:06.302975 19313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "f0c4b34935c94ea099e1841394037d53"
format_stamp: "Formatted at 2026-06-20 22:01:06 on dist-test-slave-gl2g"
server_key: "00d640bb65bbba564fd0dc5916f4d34d"
server_key_iv: "861bf55a74ccf2a6eafdf38ace14fed0"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:06.304404 19313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:06.315080 19328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:06.315294 19313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:01:06.315389 19313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f0c4b34935c94ea099e1841394037d53"
format_stamp: "Formatted at 2026-06-20 22:01:06 on dist-test-slave-gl2g"
server_key: "00d640bb65bbba564fd0dc5916f4d34d"
server_key_iv: "861bf55a74ccf2a6eafdf38ace14fed0"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:06.315470 19313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:06.325100 19313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:06.327128 19313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:06.327261 19313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:06.331722 19313 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:43371
I20260620 22:01:06.331770 19380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:43371 every 8 connection(s)
I20260620 22:01:06.332086 19313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:01:06.332871 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:06.335939 19381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Bootstrap starting.
I20260620 22:01:06.336630 19381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:06.336957 19381 log.cc:824] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:06.337707 19381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: No bootstrap required, opened a new log
I20260620 22:01:06.340003 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 19313
I20260620 22:01:06.340113 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:01:06.340301 7110 external_mini_cluster.cc:1428] Setting key 2afc6a914f91907c65faf6733cdef967
I20260620 22:01:06.340461 19381 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } }
I20260620 22:01:06.340641 19381 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:06.340672 19381 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c4b34935c94ea099e1841394037d53, State: Initialized, Role: FOLLOWER
I20260620 22:01:06.340811 19381 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } }
I20260620 22:01:06.340906 19381 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:01:06.340937 19381 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:01:06.340981 19381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:06.341620 19381 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } }
I20260620 22:01:06.341735 19381 leader_election.cc:302] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0c4b34935c94ea099e1841394037d53; no voters:
I20260620 22:01:06.341905 19381 leader_election.cc:288] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:01:06.342024 19386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:06.342133 19381 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:06.342260 19386 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 1 LEADER]: Becoming Leader. State: Replica: f0c4b34935c94ea099e1841394037d53, State: Running, Role: LEADER
I20260620 22:01:06.342401 19386 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } }
I20260620 22:01:06.342764 19386 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0c4b34935c94ea099e1841394037d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } } }
I20260620 22:01:06.342886 19386 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:06.343073 19386 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0c4b34935c94ea099e1841394037d53. Latest consensus state: current_term: 1 leader_uuid: "f0c4b34935c94ea099e1841394037d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } } }
I20260620 22:01:06.343165 19386 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:06.343291 19395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:01:06.343593 19381 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:01:06.343830 19395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:01:06 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:06.354424 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:06.340777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.6.241.190:43371)
0620 22:01:06.340951 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:06.340960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:06.341156 (+ 196us) server_negotiation.cc:407] Connection header received
0620 22:01:06.341292 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:06.341298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:06.341338 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:06.341487 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:06.342327 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:06.343761 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:06.345518 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:06.347119 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:06.347957 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:06.350063 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:06.350067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:06.350067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:06.350079 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:06.352140 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:06.352550 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:06.352553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:06.352553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:06.352669 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:06.352882 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:06.352883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:06.352883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:06.353167 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:06.353279 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:06.353800 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 22:01:06.353935 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260620 22:01:06.363194 19395 catalog_manager.cc:1347] Generated new cluster ID: 6371d4c264bb421cbd629535bf73c36d
I20260620 22:01:06.363250 19395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:01:06.384987 19395 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:01:06.385699 19395 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:01:06.391161 19395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Generated new TSK 0
I20260620 22:01:06.391322 19395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 22:01:07.057351 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:60133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43371
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:07.155198 19426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:07.155381 19426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:07.155403 19426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:07.156636 19426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:07.156677 19426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:07.156704 19426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:01:07.158236 19426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.241.148:60133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:07.158447 19426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:07.158758 19426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:07.165038 19434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:07.165059 19431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:07.165367 19432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:07.165843 19426 server_base.cc:1034] running on GCE node
I20260620 22:01:07.166040 19426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:07.166298 19426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:07.167483 19426 hybrid_clock.cc:648] HybridClock initialized: now 1781992867167465 us; error 31 us; skew 500 ppm
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:07.170464 19426 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:01:07.170962 19426 webserver.cc:460] Webserver started at http://127.6.241.129:38975/ using document root <none> and password file <none>
I20260620 22:01:07.171180 19426 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:07.171248 19426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:07.171391 19426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:07.184584 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "bbb3248579294c23a2c6e5bb91a470cf"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "bb1a90bd8ccec93c23167dfda2e235d6"
server_key_iv: "1e7aff1fb9b4fca2de0c409a5a75422c"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.185091 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "bbb3248579294c23a2c6e5bb91a470cf"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "bb1a90bd8ccec93c23167dfda2e235d6"
server_key_iv: "1e7aff1fb9b4fca2de0c409a5a75422c"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.190984 19426 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260620 22:01:07.205224 19443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.205428 19426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:07.205511 19426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "bbb3248579294c23a2c6e5bb91a470cf"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "bb1a90bd8ccec93c23167dfda2e235d6"
server_key_iv: "1e7aff1fb9b4fca2de0c409a5a75422c"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.205600 19426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:07.232759 19426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:07.235066 19426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:07.235226 19426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:07.235458 19426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:07.235818 19426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:07.235874 19426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.235916 19426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:07.235946 19426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.241873 19426 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:33247
I20260620 22:01:07.242326 19426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:01:07.244789 19556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:33247 every 8 connection(s)
I20260620 22:01:07.247268 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 19426
I20260620 22:01:07.247383 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:01:07.247565 7110 external_mini_cluster.cc:1428] Setting key 9130ba97a6e4e316093c57d788c81ffc
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:07.262385 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:07.245530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43371 (local address 127.6.241.129:60711)
0620 22:01:07.246115 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:07.246154 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:07.246807 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:07.247205 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:07.247209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:07.247589 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:07.247892 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:07.247899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.248924 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.248925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:07.250169 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:07.250172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.250742 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.250743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:07.250872 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:07.257181 (+ 6309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:07.257191 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:07.258840 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:07.260850 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:07.260857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:07.260859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:07.261140 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:07.261351 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:07.261351 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:07.261352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:07.261482 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:07.261767 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:07.261772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:07.261920 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 22:01:07.262017 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 22:01:07.262160 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":73,"threads_started":1}
I20260620 22:01:07.262921 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:07.245324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:60711 (local address 127.6.241.190:43371)
0620 22:01:07.246979 (+ 1655us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:07.246982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:07.246989 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:01:07.247015 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:07.247015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:07.247028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:07.247088 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:07.248001 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.248817 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.250264 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.250580 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.250662 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:07.258952 (+ 8290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:07.258953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:07.258953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:07.258961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:07.260757 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:07.261223 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:07.261224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:07.261225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:07.261287 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:07.261559 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:07.261560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:07.261560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:07.261691 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:07.261744 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:07.262747 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0620 22:01:07.262831 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1627,"thread_start_us":82,"threads_started":1}
I20260620 22:01:07.263619 19557 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43371
I20260620 22:01:07.263707 19557 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:07.263877 19557 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
I20260620 22:01:07.264391 19345 ts_manager.cc:194] Registered new tserver with Master: bbb3248579294c23a2c6e5bb91a470cf (127.6.241.129:33247)
I20260620 22:01:07.265239 19345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:60711
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 22:01:07.289580 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:60133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43371
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 22:01:07.406776 19565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:07.407073 19565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:07.407112 19565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:07.408712 19565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:07.408764 19565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:07.408802 19565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:01:07.410902 19565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.241.148:60133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:07.411137 19565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:07.411418 19565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:07.416950 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:07.416972 19570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:07.416954 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:07.418004 19565 server_base.cc:1034] running on GCE node
I20260620 22:01:07.418215 19565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:07.418458 19565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:07.419667 19565 hybrid_clock.cc:648] HybridClock initialized: now 1781992867419651 us; error 33 us; skew 500 ppm
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:07.422770 19565 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:01:07.423250 19565 webserver.cc:460] Webserver started at http://127.6.241.130:37733/ using document root <none> and password file <none>
I20260620 22:01:07.423491 19565 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:07.423561 19565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:07.423743 19565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:07.436631 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "3a7153ee9ac54b9a94a7886c665dec3e"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "f09cc9ec5418e955c693cfd1ba72dd78"
server_key_iv: "099acd8afd2649b94e19617abcf9ad87"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.437331 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "3a7153ee9ac54b9a94a7886c665dec3e"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "f09cc9ec5418e955c693cfd1ba72dd78"
server_key_iv: "099acd8afd2649b94e19617abcf9ad87"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.441910 19565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 22:01:07.456727 19581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.456962 19565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:01:07.457051 19565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "3a7153ee9ac54b9a94a7886c665dec3e"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "f09cc9ec5418e955c693cfd1ba72dd78"
server_key_iv: "099acd8afd2649b94e19617abcf9ad87"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.457139 19565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:07.483914 19565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:07.486377 19565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:07.486557 19565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:07.486814 19565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:07.487195 19565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:07.487270 19565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.487334 19565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:07.487397 19565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.493956 19565 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:39337
I20260620 22:01:07.494429 19565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:01:07.494695 19694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:39337 every 8 connection(s)
I20260620 22:01:07.496121 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 19565
I20260620 22:01:07.496215 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:01:07.496443 7110 external_mini_cluster.cc:1428] Setting key dab6e3c67e32c37fecb9e5fb9058f752
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:07.508973 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:07.495242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43371 (local address 127.6.241.130:54651)
0620 22:01:07.495707 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:07.495752 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:07.497157 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:07.497526 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:07.497544 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:07.497766 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:07.498023 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:07.498030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.499049 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.499050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:07.500301 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:07.500303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.500878 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.500879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:07.501032 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:07.501563 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:07.501572 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:07.503245 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:07.507284 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:07.507290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:07.507292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:07.507589 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:07.507810 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:07.507811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:07.507812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:07.507969 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:07.508285 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:07.508291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:07.508472 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 22:01:07.508583 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 22:01:07.508734 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":82,"threads_started":1}
I20260620 22:01:07.509500 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:07.495490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:54651 (local address 127.6.241.190:43371)
0620 22:01:07.495586 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:07.495589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:07.495879 (+ 290us) server_negotiation.cc:407] Connection header received
0620 22:01:07.497279 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:07.497281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:07.497294 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:07.497323 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:07.498134 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.498940 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.500403 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.500736 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.500781 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:07.503344 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:07.503345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:07.503346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:07.503353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:07.507197 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:07.507678 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:07.507679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:07.507679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:07.507738 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:07.508057 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:07.508058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:07.508059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:07.508193 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:07.508256 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:07.509319 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0620 22:01:07.509406 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 22:01:07.510111 19695 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43371
I20260620 22:01:07.510201 19695 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:07.510370 19695 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
I20260620 22:01:07.510778 19345 ts_manager.cc:194] Registered new tserver with Master: 3a7153ee9ac54b9a94a7886c665dec3e (127.6.241.130:39337)
I20260620 22:01:07.511286 19345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:54651
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 22:01:07.532743 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:60133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43371
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:07.640156 19700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:07.640486 19700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:07.640576 19700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:07.642194 19700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:07.642288 19700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:07.642364 19700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:01:07.644480 19700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.241.148:60133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:07.644908 19700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:07.645191 19700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:07.651891 19708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:07.652129 19700 server_base.cc:1034] running on GCE node
W20260620 22:01:07.652226 19705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:07.652354 19706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:07.652626 19700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:07.652915 19700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:07.654083 19700 hybrid_clock.cc:648] HybridClock initialized: now 1781992867654076 us; error 30 us; skew 500 ppm
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:07.656579 19700 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:01:07.657073 19700 webserver.cc:460] Webserver started at http://127.6.241.131:44373/ using document root <none> and password file <none>
I20260620 22:01:07.657290 19700 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:07.657358 19700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:07.657531 19700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:07.669514 19700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "da21d65e382942f6b856bef133b8530d"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "5167cf6deaa72a95c0537d4211a0ade8"
server_key_iv: "b314433f701d478773b30f309a04370a"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.669936 19700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "da21d65e382942f6b856bef133b8530d"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "5167cf6deaa72a95c0537d4211a0ade8"
server_key_iv: "b314433f701d478773b30f309a04370a"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.671965 19700 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 22:01:07.698741 19715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.698966 19700 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.000s
I20260620 22:01:07.699059 19700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "da21d65e382942f6b856bef133b8530d"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "5167cf6deaa72a95c0537d4211a0ade8"
server_key_iv: "b314433f701d478773b30f309a04370a"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.699158 19700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:07.726826 19700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:07.729141 19700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:07.729288 19700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:07.729550 19700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:07.729921 19700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:07.729983 19700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.730022 19700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:07.730047 19700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.737156 19700 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:42613
I20260620 22:01:07.737988 19700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:01:07.738478 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 19700
I20260620 22:01:07.738603 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:01:07.738813 7110 external_mini_cluster.cc:1428] Setting key 7b4de547c08d00bfea7957683b8a87c2
I20260620 22:01:07.746260 19828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:42613 every 8 connection(s)
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:07.758083 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:07.746834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:38277 (local address 127.6.241.190:43371)
0620 22:01:07.746878 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:07.746880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:07.750147 (+ 3267us) server_negotiation.cc:407] Connection header received
0620 22:01:07.750938 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:07.750939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:07.750950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:07.750981 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:07.751747 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.752221 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.753197 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.753390 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.753463 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:07.755417 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:07.755419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:07.755419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:07.755427 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:07.756792 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:07.757179 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:07.757180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:07.757180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:07.757238 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:07.757476 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:07.757477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:07.757477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:07.757586 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:07.757658 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:07.757912 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 22:01:07.757986 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:01:07.758260 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:07.746849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43371 (local address 127.6.241.131:38277)
0620 22:01:07.750007 (+ 3158us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:07.750057 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:07.750820 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:07.751179 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:07.751184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:07.751421 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:07.751641 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:07.751646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.752316 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.752317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:07.753082 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:07.753084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:07.753481 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:07.753482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:07.753601 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:07.754114 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:07.754122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:07.755330 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:07.756887 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:07.756893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:07.756895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:07.757105 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:07.757302 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:07.757303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:07.757303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:07.757410 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:07.757650 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:07.757655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:07.757787 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 22:01:07.757892 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 22:01:07.758027 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3120,"thread_start_us":78,"threads_started":1}
I20260620 22:01:07.758749 19829 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43371
I20260620 22:01:07.758837 19829 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:07.759006 19829 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
I20260620 22:01:07.759373 19345 ts_manager.cc:194] Registered new tserver with Master: da21d65e382942f6b856bef133b8530d (127.6.241.131:42613)
I20260620 22:01:07.759755 19345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:38277
I20260620 22:01:07.761319 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:01:07.761970 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 19426
I20260620 22:01:07.768342 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 19565
I20260620 22:01:07.772820 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 19700
I20260620 22:01:07.781791 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 19313
I20260620 22:01:07.788205 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:43371
--webserver_interface=127.6.241.190
--webserver_port=35571
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:43371
--ranger_config_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:60133/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.241.148:60133/kms with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:07.867902 19832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:07.868091 19832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:07.868111 19832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:07.869160 19832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:07.869194 19832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:07.869207 19832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:07.869220 19832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:07.869232 19832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:07.870678 19832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.241.148:60133/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:43371
--ranger_config_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=35571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:07.870844 19832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:07.871009 19832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:07.875496 19840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:07.875546 19838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:07.875505 19837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:07.875741 19832 server_base.cc:1034] running on GCE node
I20260620 22:01:07.875976 19832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:07.876250 19832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:07.877410 19832 hybrid_clock.cc:648] HybridClock initialized: now 1781992867877396 us; error 25 us; skew 500 ppm
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:07.879817 19832 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:01:07.880254 19832 webserver.cc:460] Webserver started at http://127.6.241.190:35571/ using document root <none> and password file <none>
I20260620 22:01:07.880470 19832 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:07.880548 19832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:01:07 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:07.922816 19832 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:07.924279 19848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:07.924523 19832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:07.924644 19832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f0c4b34935c94ea099e1841394037d53"
format_stamp: "Formatted at 2026-06-20 22:01:06 on dist-test-slave-gl2g"
server_key: "00d640bb65bbba564fd0dc5916f4d34d"
server_key_iv: "861bf55a74ccf2a6eafdf38ace14fed0"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:07.925026 19832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:07.938512 19832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:07.939996 19832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:07.940147 19832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:07.944209 19832 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:43371
I20260620 22:01:07.944242 19900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:43371 every 8 connection(s)
I20260620 22:01:07.944602 19832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:01:07.945765 19901 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:01:07.946393 19901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Bootstrap starting.
I20260620 22:01:07.953774 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 19832
I20260620 22:01:07.954106 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:60133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:33247
--local_ip_for_outbound_sockets=127.6.241.129
--tserver_master_addrs=127.6.241.190:43371
--webserver_port=38975
--webserver_interface=127.6.241.129
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.241.148:60133/kms with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:07.954125 19901 log.cc:824] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:07.955840 19901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:07.956135 19901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Bootstrap complete.
I20260620 22:01:07.958556 19901 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } }
I20260620 22:01:07.958869 19901 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c4b34935c94ea099e1841394037d53, State: Initialized, Role: FOLLOWER
I20260620 22:01:07.959074 19901 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } }
I20260620 22:01:07.959180 19901 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:01:07.959231 19901 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:01:07.959278 19901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:01:07.960314 19901 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } }
I20260620 22:01:07.960427 19901 leader_election.cc:302] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0c4b34935c94ea099e1841394037d53; no voters:
I20260620 22:01:07.960619 19901 leader_election.cc:288] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:01:07.960650 19905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:01:07.960750 19905 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [term 2 LEADER]: Becoming Leader. State: Replica: f0c4b34935c94ea099e1841394037d53, State: Running, Role: LEADER
I20260620 22:01:07.960835 19905 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } }
I20260620 22:01:07.960865 19901 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:07.961104 19905 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0c4b34935c94ea099e1841394037d53. Latest consensus state: current_term: 2 leader_uuid: "f0c4b34935c94ea099e1841394037d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } } }
I20260620 22:01:07.961182 19905 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:07.961108 19907 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f0c4b34935c94ea099e1841394037d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c4b34935c94ea099e1841394037d53" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 43371 } } }
I20260620 22:01:07.961230 19907 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:07.962265 19901 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:01:07.962594 19909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:01:07.964949 19909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:01:07.965194 19909 catalog_manager.cc:1259] Loaded cluster ID: 6371d4c264bb421cbd629535bf73c36d
I20260620 22:01:07.965240 19909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:01:07.966881 19909 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:01:07.967095 19909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Loaded TSK: 0
I20260620 22:01:07.967327 19909 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:01:08.042766 19904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:08.042970 19904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:08.043009 19904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:08.044359 19904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:08.044430 19904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:08.044473 19904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:01:08.046025 19904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.241.148:60133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:33247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=38975
--tserver_master_addrs=127.6.241.190:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:08.046276 19904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:08.046480 19904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:08.051460 19941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:08.051472 19938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:08.051702 19939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:08.052057 19904 server_base.cc:1034] running on GCE node
I20260620 22:01:08.052256 19904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:08.052518 19904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:08.055763 19904 hybrid_clock.cc:648] HybridClock initialized: now 1781992868055747 us; error 25 us; skew 500 ppm
Jun 20 22:01:08 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:08.058408 19904 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:01:08.059070 19904 webserver.cc:460] Webserver started at http://127.6.241.129:38975/ using document root <none> and password file <none>
I20260620 22:01:08.059317 19904 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:08.059382 19904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:01:08 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:08.085843 19904 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:08.095249 19949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:08.095469 19904 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260620 22:01:08.095566 19904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "bbb3248579294c23a2c6e5bb91a470cf"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "bb1a90bd8ccec93c23167dfda2e235d6"
server_key_iv: "1e7aff1fb9b4fca2de0c409a5a75422c"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:08.096024 19904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:08.118839 19904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:08.121255 19904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:08.121425 19904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:08.121680 19904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:08.122033 19904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:08.122082 19904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:08.122130 19904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:08.122153 19904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:08.129328 19904 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:33247
I20260620 22:01:08.129781 19904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:01:08.130036 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 19904
I20260620 22:01:08.130327 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:60133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:39337
--local_ip_for_outbound_sockets=127.6.241.130
--tserver_master_addrs=127.6.241.190:43371
--webserver_port=37733
--webserver_interface=127.6.241.130
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.241.148:60133/kms with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:08.132987 20062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:33247 every 8 connection(s)
Jun 20 22:01:08 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:08.146452 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:08.133601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43371 (local address 127.6.241.129:49395)
0620 22:01:08.135220 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:08.135259 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:08.135807 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:08.136366 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:08.136371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:08.136603 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:08.136863 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:08.136869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.138544 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.138545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:08.139704 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:08.139706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.140632 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.140633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:08.140760 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:08.141286 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:08.141295 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:08.142624 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:08.144654 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:08.144659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:08.144661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:08.144925 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:08.145206 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:08.145207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:08.145208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:08.145335 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:08.145821 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:08.145826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:08.145973 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 22:01:08.146085 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 22:01:08.146227 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1582,"thread_start_us":97,"threads_started":1}
I20260620 22:01:08.147414 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:08.133659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:49395 (local address 127.6.241.190:43371)
0620 22:01:08.135136 (+ 1477us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:08.135145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:08.135943 (+ 798us) server_negotiation.cc:407] Connection header received
0620 22:01:08.136084 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:08.136090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:08.136139 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:08.136254 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:08.137015 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.138425 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.139795 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.140430 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.140530 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:08.142743 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:08.142746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:08.142747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:08.142757 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:08.144568 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:08.145009 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:08.145011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:08.145012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:08.145130 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:08.145410 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:08.145411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:08.145411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:08.145716 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:08.145791 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:08.147103 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0620 22:01:08.147210 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1424,"thread_start_us":65,"threads_started":1}
I20260620 22:01:08.148414 20063 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43371
I20260620 22:01:08.148497 20063 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:08.148662 20063 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
W20260620 22:01:08.149520 20063 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:43371 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:01:08.149580 20063 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:08.149638 20063 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
I20260620 22:01:08.150020 20063 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:08.150090 20063 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
W20260620 22:01:08.150398 20063 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:01:08.242831 20065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:08.243095 20065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:08.243134 20065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:08.244359 20065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:08.244426 20065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:08.244468 20065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:01:08.246713 20065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.241.148:60133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:39337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=37733
--tserver_master_addrs=127.6.241.190:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:08.247112 20065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:08.247426 20065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:08.253705 20075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:08.253999 20073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:08.254761 20072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:08.256220 20065 server_base.cc:1034] running on GCE node
I20260620 22:01:08.256464 20065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:08.256739 20065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:08.257912 20065 hybrid_clock.cc:648] HybridClock initialized: now 1781992868257896 us; error 34 us; skew 500 ppm
Jun 20 22:01:08 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:08.260802 20065 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:01:08.261325 20065 webserver.cc:460] Webserver started at http://127.6.241.130:37733/ using document root <none> and password file <none>
I20260620 22:01:08.261693 20065 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:08.261773 20065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:01:08 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:08.276170 20065 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:08.283592 20083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:08.283849 20065 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.000s
I20260620 22:01:08.283996 20065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "3a7153ee9ac54b9a94a7886c665dec3e"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "f09cc9ec5418e955c693cfd1ba72dd78"
server_key_iv: "099acd8afd2649b94e19617abcf9ad87"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:08.285116 20065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:08.316835 20065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:08.319033 20065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:08.319171 20065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:08.319437 20065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:08.319819 20065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:08.319880 20065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:08.319903 20065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:08.319916 20065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:08.327288 20065 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:39337
I20260620 22:01:08.327759 20065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:01:08.328985 20196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:39337 every 8 connection(s)
I20260620 22:01:08.330511 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 20065
I20260620 22:01:08.330818 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.241.148:60133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:42613
--local_ip_for_outbound_sockets=127.6.241.131
--tserver_master_addrs=127.6.241.190:43371
--webserver_port=44373
--webserver_interface=127.6.241.131
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.241.148:60133/kms with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:01:08 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:08.340188 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:08.329947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:47761 (local address 127.6.241.190:43371)
0620 22:01:08.330019 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:08.330021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:08.330031 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:01:08.330758 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:08.330759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:08.330772 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:08.330805 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:08.331584 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.332311 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.333746 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.334035 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.334110 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:08.336747 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:08.336748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:08.336749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:08.336756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:08.338539 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:08.339021 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:08.339022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:08.339023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:08.339092 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:08.339398 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:08.339399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:08.339400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:08.339556 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:08.339632 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:08.339985 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 22:01:08.340084 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:01:08.340888 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:08.329649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43371 (local address 127.6.241.130:47761)
0620 22:01:08.329848 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:08.329910 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:08.330626 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:08.330997 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:08.331001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:08.331220 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:08.331478 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:08.331483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.332428 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.332429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:08.333635 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:08.333638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.334190 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.334191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:08.334340 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:08.334930 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:08.334939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:08.336643 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:08.338645 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:08.338651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:08.338653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:08.338925 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:08.339166 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:08.339167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:08.339167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:08.339294 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:08.339636 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:08.339643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:08.339793 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 22:01:08.340508 (+ 715us) client_negotiation.cc:240] Negotiation successful
0620 22:01:08.340662 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260620 22:01:08.341825 20197 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43371
I20260620 22:01:08.341964 20197 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:08.342141 20197 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
W20260620 22:01:08.342818 20197 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:43371 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:01:08.342890 20197 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:08.342945 20197 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
I20260620 22:01:08.343278 20197 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:08.343349 20197 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
W20260620 22:01:08.343623 20197 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:01:08.435855 20200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:08.436241 20200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:08.436275 20200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:08.437932 20200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:08.438032 20200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:08.438098 20200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:01:08.440165 20200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.241.148:60133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:42613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=44373
--tserver_master_addrs=127.6.241.190:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:08.440570 20200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:08.440842 20200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:08.453789 20209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:08.453795 20207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:08.454106 20206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:08.455070 20200 server_base.cc:1034] running on GCE node
I20260620 22:01:08.455268 20200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:08.455528 20200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:08.456692 20200 hybrid_clock.cc:648] HybridClock initialized: now 1781992868456673 us; error 35 us; skew 500 ppm
Jun 20 22:01:08 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:08.459841 20200 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:01:08.460330 20200 webserver.cc:460] Webserver started at http://127.6.241.131:44373/ using document root <none> and password file <none>
I20260620 22:01:08.460637 20200 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:08.460716 20200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:01:08 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:08.477923 20200 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:01:08.483240 20217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:08.483441 20200 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 22:01:08.483520 20200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "da21d65e382942f6b856bef133b8530d"
format_stamp: "Formatted at 2026-06-20 22:01:07 on dist-test-slave-gl2g"
server_key: "5167cf6deaa72a95c0537d4211a0ade8"
server_key_iv: "b314433f701d478773b30f309a04370a"
server_key_version: "kuduclusterkey@0"
I20260620 22:01:08.483960 20200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:08.508411 20200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:08.512152 20200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:08.512352 20200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:08.512630 20200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:08.513018 20200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:08.513077 20200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:08.513115 20200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:08.513141 20200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:08.520645 20200 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:42613
I20260620 22:01:08.520741 20330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:42613 every 8 connection(s)
I20260620 22:01:08.521108 20200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:01:08 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:08.529326 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 20200
I20260620 22:01:08.531133 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:08.521839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:51589 (local address 127.6.241.190:43371)
0620 22:01:08.521885 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:08.521887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:08.522159 (+ 272us) server_negotiation.cc:407] Connection header received
0620 22:01:08.522718 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:08.522719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:08.522733 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:08.522796 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:08.523455 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.524213 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.525297 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.525488 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.525570 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:08.527757 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:08.527759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:08.527759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:08.527767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:08.529450 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:08.529974 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:08.529975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:08.529975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:08.530040 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:08.530328 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:08.530329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:08.530329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:08.530481 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:08.530532 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:08.530942 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 22:01:08.531027 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:01:08.531635 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:08.521828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:43371 (local address 127.6.241.131:51589)
0620 22:01:08.521993 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:08.522039 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:08.522585 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:08.522869 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:08.522874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:08.523086 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:08.523344 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:08.523351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.524308 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.524309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:08.525187 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:08.525190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.525595 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.525597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:08.525706 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:08.526171 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:08.526196 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:08.527669 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:08.529546 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:08.529552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:08.529554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:08.529879 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:08.530118 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:08.530119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:08.530119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:08.530246 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:08.530586 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:08.530592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:08.530773 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 22:01:08.531242 (+ 469us) client_negotiation.cc:240] Negotiation successful
0620 22:01:08.531408 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260620 22:01:08.532162 20331 heartbeater.cc:347] Connected to a master server at 127.6.241.190:43371
I20260620 22:01:08.532244 20331 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:08.532459 20331 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
W20260620 22:01:08.533151 20331 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:43371 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:01:08.533277 20331 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:08.533375 20331 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
I20260620 22:01:08.533725 20331 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:08.533836 20331 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
W20260620 22:01:08.534271 20331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:01:08.542095 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:08.532066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.6.241.190:43371)
0620 22:01:08.532097 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:08.532099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:08.535502 (+ 3403us) server_negotiation.cc:407] Connection header received
0620 22:01:08.535582 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:08.535583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:08.535592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:08.535646 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:08.536211 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.536672 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.537406 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:08.537601 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:08.537712 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:08.539130 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:08.539131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:08.539131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:08.539137 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:08.540792 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:08.541196 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:08.541197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:08.541197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:08.541247 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:08.541433 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:08.541434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:08.541434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:08.541562 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:08.541705 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:08.541939 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 22:01:08.542012 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
Jun 20 22:01:09 dist-test-slave-gl2g krb5kdc[18869](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:09.150605 20063 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:09.150751 20063 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
I20260620 22:01:09.151140 19865 ts_manager.cc:194] Registered new tserver with Master: bbb3248579294c23a2c6e5bb91a470cf (127.6.241.129:33247)
I20260620 22:01:09.152093 19865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:49395
I20260620 22:01:09.343804 20197 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:09.343967 20197 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
I20260620 22:01:09.344228 19865 ts_manager.cc:194] Registered new tserver with Master: 3a7153ee9ac54b9a94a7886c665dec3e (127.6.241.130:39337)
I20260620 22:01:09.344763 19865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:47761
Jun 20 22:01:09 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:09.534482 20331 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:09.534638 20331 heartbeater.cc:511] Master 127.6.241.190:43371 requested a full tablet report, sending...
I20260620 22:01:09.534914 19865 ts_manager.cc:194] Registered new tserver with Master: da21d65e382942f6b856bef133b8530d (127.6.241.131:42613)
I20260620 22:01:09.535406 19865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:51589
I20260620 22:01:09.538573 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:01:09.547179 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.539565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.6.241.190:43371)
0620 22:01:09.539709 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:09.539711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:09.539860 (+ 149us) server_negotiation.cc:407] Connection header received
0620 22:01:09.539920 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:09.539921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:09.539968 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:09.540046 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:09.540761 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.541497 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.542497 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.542809 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.542872 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.544324 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:09.544325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:09.544325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:09.544331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:09.546004 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:09.546383 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:09.546384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:09.546384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:09.546447 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:09.546656 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:09.546657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:09.546657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:09.546784 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:09.546877 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:09.547011 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 22:01:09.547085 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260620 22:01:09.549216 19865 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:01:09.551447 19865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:01:09.559141 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.554287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:33247 (local address 127.0.0.1:58840)
0620 22:01:09.554428 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:09.554466 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:09.554529 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:09.554980 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:09.554983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:09.554985 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:09.555047 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.555048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.556327 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.556328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.557828 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.557831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.558858 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.558859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.558941 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.558955 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:01:09.559003 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:01:09.559051 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260620 22:01:09.559758 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.554662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.6.241.129:33247)
0620 22:01:09.554806 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:09.554810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:09.554821 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:01:09.554854 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:09.554856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:09.554871 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:09.555118 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:09.555190 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.556236 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.557924 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.558736 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.559545 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.559616 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:09.559643 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 22:01:09.559685 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
I20260620 22:01:09.560546 19997 tablet_service.cc:1467] Processing CreateTablet for tablet 99444f4762bc4e6abc918a1eb4d8edb6 (DEFAULT_TABLE table=test-table [id=018487816a704b6582d4d126cd33fe81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:09.560930 19997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99444f4762bc4e6abc918a1eb4d8edb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:09.565033 20363 tablet_bootstrap.cc:492] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf: Bootstrap starting.
I20260620 22:01:09.565685 20363 tablet_bootstrap.cc:654] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:09.566023 20363 log.cc:824] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:09.566905 20363 tablet_bootstrap.cc:492] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf: No bootstrap required, opened a new log
I20260620 22:01:09.567054 20363 ts_tablet_manager.cc:1397] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:01:09.569360 20363 raft_consensus.cc:348] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } }
I20260620 22:01:09.569630 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.554496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:39337 (local address 127.0.0.1:58476)
0620 22:01:09.555472 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:09.555482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:09.555524 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:09.565527 (+ 10003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:09.565528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:09.565531 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:09.565571 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.565572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.566684 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.566685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.567911 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.567914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.569393 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.569394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.569454 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.569459 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:01:09.569511 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:01:09.569551 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":66,"threads_started":1}
I20260620 22:01:09.569629 20363 raft_consensus.cc:374] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:09.569674 20363 raft_consensus.cc:729] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb3248579294c23a2c6e5bb91a470cf, State: Initialized, Role: FOLLOWER
I20260620 22:01:09.569898 20363 consensus_queue.cc:260] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } }
I20260620 22:01:09.570005 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.554919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.6.241.130:39337)
0620 22:01:09.562708 (+ 7789us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:09.562713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:09.562724 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:01:09.562758 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:09.562760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:09.562776 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:09.562858 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:09.565648 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.566555 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.568823 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.569304 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.569394 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.569451 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:09.569876 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 22:01:09.569929 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7750,"thread_start_us":52,"threads_started":1}
I20260620 22:01:09.570236 20363 ts_tablet_manager.cc:1428] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:01:09.570420 20063 heartbeater.cc:503] Master 127.6.241.190:43371 was elected leader, sending a full tablet report...
I20260620 22:01:09.570567 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.555171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:42613 (local address 127.0.0.1:41510)
0620 22:01:09.555592 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:09.555599 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:09.555634 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:09.565657 (+ 10023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:09.565658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:09.565659 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:09.565687 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.565688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.567924 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.567924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.568758 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.568760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.570381 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.570382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.570446 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.570450 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:01:09.570479 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 22:01:09.570516 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":64,"threads_started":1}
I20260620 22:01:09.571075 20131 tablet_service.cc:1467] Processing CreateTablet for tablet 99444f4762bc4e6abc918a1eb4d8edb6 (DEFAULT_TABLE table=test-table [id=018487816a704b6582d4d126cd33fe81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:09.571453 20131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99444f4762bc4e6abc918a1eb4d8edb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:09.573982 20366 tablet_bootstrap.cc:492] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: Bootstrap starting.
I20260620 22:01:09.574285 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.555255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.6.241.131:42613)
0620 22:01:09.562984 (+ 7729us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:09.562989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:09.562998 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:01:09.563031 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:09.563033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:09.563047 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:09.563114 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:09.566715 (+ 3601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.567798 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.569511 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.570282 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.574061 (+ 3779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.574128 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:09.574156 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 22:01:09.574210 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7695,"thread_start_us":47,"threads_started":1}
I20260620 22:01:09.575091 20366 tablet_bootstrap.cc:654] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:09.575007 20265 tablet_service.cc:1467] Processing CreateTablet for tablet 99444f4762bc4e6abc918a1eb4d8edb6 (DEFAULT_TABLE table=test-table [id=018487816a704b6582d4d126cd33fe81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:09.575335 20265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99444f4762bc4e6abc918a1eb4d8edb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:09.575502 20366 log.cc:824] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:09.578444 20368 tablet_bootstrap.cc:492] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: Bootstrap starting.
I20260620 22:01:09.579177 20366 tablet_bootstrap.cc:492] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: No bootstrap required, opened a new log
I20260620 22:01:09.579269 20366 ts_tablet_manager.cc:1397] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 22:01:09.581274 20366 raft_consensus.cc:348] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } }
I20260620 22:01:09.581421 20366 raft_consensus.cc:374] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:09.581480 20366 raft_consensus.cc:729] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7153ee9ac54b9a94a7886c665dec3e, State: Initialized, Role: FOLLOWER
I20260620 22:01:09.581617 20366 consensus_queue.cc:260] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } }
I20260620 22:01:09.581947 20197 heartbeater.cc:503] Master 127.6.241.190:43371 was elected leader, sending a full tablet report...
I20260620 22:01:09.582202 20366 ts_tablet_manager.cc:1428] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:01:09.582528 20368 tablet_bootstrap.cc:654] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:09.582911 20368 log.cc:824] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:09.583623 20368 tablet_bootstrap.cc:492] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: No bootstrap required, opened a new log
I20260620 22:01:09.583705 20368 ts_tablet_manager.cc:1397] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 22:01:09.585348 20368 raft_consensus.cc:348] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } }
I20260620 22:01:09.585528 20368 raft_consensus.cc:374] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:09.585599 20368 raft_consensus.cc:729] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da21d65e382942f6b856bef133b8530d, State: Initialized, Role: FOLLOWER
I20260620 22:01:09.585713 20368 consensus_queue.cc:260] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } }
I20260620 22:01:09.585958 20368 ts_tablet_manager.cc:1428] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:01:09.586063 20331 heartbeater.cc:503] Master 127.6.241.190:43371 was elected leader, sending a full tablet report...
I20260620 22:01:09.598205 20371 raft_consensus.cc:482] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:01:09.598344 20371 raft_consensus.cc:504] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } }
I20260620 22:01:09.598702 20371 leader_election.cc:288] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbb3248579294c23a2c6e5bb91a470cf (127.6.241.129:33247), 3a7153ee9ac54b9a94a7886c665dec3e (127.6.241.130:39337)
I20260620 22:01:09.604562 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.598953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:59947 (local address 127.6.241.129:33247)
0620 22:01:09.598999 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:09.599001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:09.599315 (+ 314us) server_negotiation.cc:407] Connection header received
0620 22:01:09.599381 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:09.599382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:09.599394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:09.599450 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:09.599664 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.600492 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.602567 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.603129 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.603181 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.603190 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:09.604452 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0620 22:01:09.604498 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:01:09.604841 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.598877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:33247 (local address 127.6.241.131:59947)
0620 22:01:09.599240 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:09.599254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:09.599326 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:09.599501 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:09.599502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:09.599504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:09.599583 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.599585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.600629 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.600629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.601936 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.601939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.603705 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.603706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.603769 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.603773 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:01:09.604720 (+ 947us) client_negotiation.cc:240] Negotiation successful
0620 22:01:09.604758 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":50,"thread_start_us":94,"threads_started":1}
I20260620 22:01:09.605289 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.599154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:35209 (local address 127.6.241.130:39337)
0620 22:01:09.599193 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:09.599195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:09.599461 (+ 266us) server_negotiation.cc:407] Connection header received
0620 22:01:09.599553 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:09.599554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:09.599564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:09.599592 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:09.599796 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.600488 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.604417 (+ 3929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.604963 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.605035 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.605042 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:09.605173 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 22:01:09.605211 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:01:09.605598 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.599099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:39337 (local address 127.6.241.131:35209)
0620 22:01:09.599414 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:09.599422 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:09.599484 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:09.599665 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:09.599665 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:09.599679 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:09.599727 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.599728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.600586 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.600587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.604330 (+ 3743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:09.604333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.605034 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.605035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:09.605093 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.605097 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:01:09.605471 (+ 374us) client_negotiation.cc:240] Negotiation successful
0620 22:01:09.605517 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":1229,"thread_start_us":62,"threads_started":1}
I20260620 22:01:09.605592 20017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99444f4762bc4e6abc918a1eb4d8edb6" candidate_uuid: "da21d65e382942f6b856bef133b8530d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb3248579294c23a2c6e5bb91a470cf" is_pre_election: true
I20260620 22:01:09.605757 20017 raft_consensus.cc:2454] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da21d65e382942f6b856bef133b8530d in term 0.
I20260620 22:01:09.606083 20219 leader_election.cc:302] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbb3248579294c23a2c6e5bb91a470cf, da21d65e382942f6b856bef133b8530d; no voters:
I20260620 22:01:09.606194 20371 raft_consensus.cc:2781] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:01:09.606263 20371 raft_consensus.cc:482] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:01:09.606307 20371 raft_consensus.cc:3037] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:09.606379 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99444f4762bc4e6abc918a1eb4d8edb6" candidate_uuid: "da21d65e382942f6b856bef133b8530d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" is_pre_election: true
I20260620 22:01:09.606495 20132 raft_consensus.cc:2454] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da21d65e382942f6b856bef133b8530d in term 0.
I20260620 22:01:09.607156 20371 raft_consensus.cc:504] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } }
I20260620 22:01:09.607303 20371 leader_election.cc:288] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [CANDIDATE]: Term 1 election: Requested vote from peers bbb3248579294c23a2c6e5bb91a470cf (127.6.241.129:33247), 3a7153ee9ac54b9a94a7886c665dec3e (127.6.241.130:39337)
I20260620 22:01:09.607457 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99444f4762bc4e6abc918a1eb4d8edb6" candidate_uuid: "da21d65e382942f6b856bef133b8530d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7153ee9ac54b9a94a7886c665dec3e"
I20260620 22:01:09.607502 20017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99444f4762bc4e6abc918a1eb4d8edb6" candidate_uuid: "da21d65e382942f6b856bef133b8530d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb3248579294c23a2c6e5bb91a470cf"
I20260620 22:01:09.607522 20132 raft_consensus.cc:3037] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:09.607553 20017 raft_consensus.cc:3037] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:09.608289 20132 raft_consensus.cc:2454] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da21d65e382942f6b856bef133b8530d in term 1.
I20260620 22:01:09.608289 20017 raft_consensus.cc:2454] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da21d65e382942f6b856bef133b8530d in term 1.
I20260620 22:01:09.608450 20219 leader_election.cc:302] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7153ee9ac54b9a94a7886c665dec3e, da21d65e382942f6b856bef133b8530d; no voters:
I20260620 22:01:09.608593 20371 raft_consensus.cc:2781] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:09.608728 20371 raft_consensus.cc:686] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 1 LEADER]: Becoming Leader. State: Replica: da21d65e382942f6b856bef133b8530d, State: Running, Role: LEADER
I20260620 22:01:09.608842 20371 consensus_queue.cc:237] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } }
I20260620 22:01:09.611563 19865 catalog_manager.cc:5495] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d reported cstate change: term changed from 0 to 1, leader changed from <none> to da21d65e382942f6b856bef133b8530d (127.6.241.131). New cstate: current_term: 1 leader_uuid: "da21d65e382942f6b856bef133b8530d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da21d65e382942f6b856bef133b8530d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 42613 } health_report { overall_health: HEALTHY } } }
I20260620 22:01:09.624010 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.620128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.6.241.131:42613)
0620 22:01:09.620166 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:09.620168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:09.620175 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:01:09.620210 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:09.620211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:09.620224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:09.620304 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:01:09.620407 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.620929 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.623020 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:09.623289 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:09.623351 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:09.623528 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:01:09.623636 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:01:09.623698 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:09.623922 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 22:01:09.623940 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 22:01:09 dist-test-slave-gl2g krb5kdc[18869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for HTTP/127.6.241.148@KRBTEST.COM
I20260620 22:01:09.633576 20132 raft_consensus.cc:1261] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [term 1 FOLLOWER]: Refusing update from remote peer da21d65e382942f6b856bef133b8530d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:09.633889 20371 consensus_queue.cc:1035] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a7153ee9ac54b9a94a7886c665dec3e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 39337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:01:09.634850 20064 tablet.cc:2369] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:01:09.637157 20017 raft_consensus.cc:1261] T 99444f4762bc4e6abc918a1eb4d8edb6 P bbb3248579294c23a2c6e5bb91a470cf [term 1 FOLLOWER]: Refusing update from remote peer da21d65e382942f6b856bef133b8530d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:09.637424 20371 consensus_queue.cc:1035] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb3248579294c23a2c6e5bb91a470cf" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 33247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:09.641110 20378 mvcc.cc:205] Tried to move back new op lower bound from 7299042794016305152 to 7299042793925443584. Current Snapshot: MvccSnapshot[applied={T|T < 7299042794016305152}]
I20260620 22:01:09.641631 20379 mvcc.cc:205] Tried to move back new op lower bound from 7299042794016305152 to 7299042793925443584. Current Snapshot: MvccSnapshot[applied={T|T < 7299042794016305152}]
I20260620 22:01:09.643783 19865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:09.643886 19865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:09.644550 19865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0c4b34935c94ea099e1841394037d53: Sending DeleteTablet for 3 replicas of tablet 99444f4762bc4e6abc918a1eb4d8edb6
I20260620 22:01:09.644974 20131 tablet_service.cc:1514] Processing DeleteTablet for tablet 99444f4762bc4e6abc918a1eb4d8edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:09 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:58476
I20260620 22:01:09.645033 19997 tablet_service.cc:1514] Processing DeleteTablet for tablet 99444f4762bc4e6abc918a1eb4d8edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:09 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:58840
I20260620 22:01:09.645182 20265 tablet_service.cc:1514] Processing DeleteTablet for tablet 99444f4762bc4e6abc918a1eb4d8edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:09 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:41510
I20260620 22:01:09.645591 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 19904
I20260620 22:01:09.650826 20386 tablet_replica.cc:330] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: stopping tablet replica
I20260620 22:01:09.650919 20386 raft_consensus.cc:2229] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:09.651018 20386 raft_consensus.cc:2258] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:09.651165 20388 tablet_replica.cc:330] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: stopping tablet replica
I20260620 22:01:09.651242 20388 raft_consensus.cc:2229] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:01:09.651336 20388 raft_consensus.cc:2258] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:09.651917 20386 ts_tablet_manager.cc:1905] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:09.653736 20388 ts_tablet_manager.cc:1905] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:09.655378 20388 ts_tablet_manager.cc:1918] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:09.655450 20388 log.cc:1196] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/99444f4762bc4e6abc918a1eb4d8edb6
I20260620 22:01:09.655727 20388 ts_tablet_manager.cc:1939] T 99444f4762bc4e6abc918a1eb4d8edb6 P da21d65e382942f6b856bef133b8530d: Deleting consensus metadata
I20260620 22:01:09.656090 19849 catalog_manager.cc:4841] TS da21d65e382942f6b856bef133b8530d (127.6.241.131:42613): tablet 99444f4762bc4e6abc918a1eb4d8edb6 (table test-table [id=018487816a704b6582d4d126cd33fe81]) successfully deleted
I20260620 22:01:09.656261 20386 ts_tablet_manager.cc:1918] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:09.656330 20386 log.cc:1196] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/99444f4762bc4e6abc918a1eb4d8edb6
I20260620 22:01:09.656652 20386 ts_tablet_manager.cc:1939] T 99444f4762bc4e6abc918a1eb4d8edb6 P 3a7153ee9ac54b9a94a7886c665dec3e: Deleting consensus metadata
I20260620 22:01:09.657085 19850 catalog_manager.cc:4841] TS 3a7153ee9ac54b9a94a7886c665dec3e (127.6.241.130:39337): tablet 99444f4762bc4e6abc918a1eb4d8edb6 (table test-table [id=018487816a704b6582d4d126cd33fe81]) successfully deleted
W20260620 22:01:09.662035 20219 connection.cc:535] client connection to 127.6.241.129:33247 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 22:01:09.662133 20219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 22:01:09.662429 19850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:33247 (error 108)
I20260620 22:01:09.662709 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 20065
I20260620 22:01:09.668586 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.668390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:33247 (local address 127.6.241.131:35189)
0620 22:01:09.668501 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:09.668544 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 22:01:09.668735 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:09.668626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:33247 (local address 127.0.0.1:58850)
0620 22:01:09.668672 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:09.668705 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:01:09.668855 19850 catalog_manager.cc:4584] TS bbb3248579294c23a2c6e5bb91a470cf (127.6.241.129:33247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99444f4762bc4e6abc918a1eb4d8edb6: Network error: Client connection negotiation failed: client connection to 127.6.241.129:33247: connect: Connection refused (error 111)
I20260620 22:01:09.669083 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 20200
I20260620 22:01:09.675206 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 19832
I20260620 22:01:09.682026 7110 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 22:01:10.027258 7110 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 22:01:10.027340 7110 mini_ranger.cc:66] Stopping Ranger...
I20260620 22:01:10.371915 7110 mini_ranger.cc:68] Stopped Ranger
2026-06-20 22:01:10.372 UTC [18899] LOG: received smart shutdown request
2026-06-20 22:01:10.375 UTC [18899] LOG: background worker "logical replication launcher" (PID 18908) exited with exit code 1
2026-06-20 22:01:10.375 UTC [18903] LOG: shutting down
2026-06-20 22:01:10.533 UTC [18899] LOG: database system is shut down
2026-06-20T22:01:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44391 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:01:10 dist-test-slave-gl2g krb5kdc[20400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:01:10 dist-test-slave-gl2g krb5kdc[20400](info): set up 2 sockets
Jun 20 22:01:10 dist-test-slave-gl2g krb5kdc[20400](info): commencing operation
krb5kdc: starting...
W20260620 22:01:11.658567 7110 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:01:11 dist-test-slave-gl2g krb5kdc[20400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:01:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:01:11.768862 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33131
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:33131,127.6.241.189:39675,127.6.241.188:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:11.846532 20413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:11.846735 20413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:11.846755 20413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:11.847775 20413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:11.847805 20413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:11.847819 20413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:11.847831 20413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:11.847843 20413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:11.849180 20413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:33131,127.6.241.189:39675,127.6.241.188:37829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:11.849392 20413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:11.849551 20413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:11.854020 20418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:11.854034 20419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:11.854061 20421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:11.854110 20413 server_base.cc:1034] running on GCE node
I20260620 22:01:11.854393 20413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:11.854709 20413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:11.855863 20413 hybrid_clock.cc:648] HybridClock initialized: now 1781992871855856 us; error 37 us; skew 500 ppm
Jun 20 22:01:11 dist-test-slave-gl2g krb5kdc[20400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:11.858126 20413 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:01:11.858670 20413 webserver.cc:460] Webserver started at http://127.6.241.190:35969/ using document root <none> and password file <none>
I20260620 22:01:11.858899 20413 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:11.858963 20413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:11.859097 20413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:11.860435 20413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "ee3fe1c4b8ee40dfa80451360cc9d920"
format_stamp: "Formatted at 2026-06-20 22:01:11 on dist-test-slave-gl2g"
server_key: "30e0823eb6cf66694e6d1e6599187fcb"
server_key_iv: "61964eee91e5892e990d564365d4e238"
server_key_version: "encryptionkey@0"
I20260620 22:01:11.860785 20413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "ee3fe1c4b8ee40dfa80451360cc9d920"
format_stamp: "Formatted at 2026-06-20 22:01:11 on dist-test-slave-gl2g"
server_key: "30e0823eb6cf66694e6d1e6599187fcb"
server_key_iv: "61964eee91e5892e990d564365d4e238"
server_key_version: "encryptionkey@0"
I20260620 22:01:11.862036 20413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:01:11.862766 20428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:11.862926 20413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:01:11.863024 20413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "ee3fe1c4b8ee40dfa80451360cc9d920"
format_stamp: "Formatted at 2026-06-20 22:01:11 on dist-test-slave-gl2g"
server_key: "30e0823eb6cf66694e6d1e6599187fcb"
server_key_iv: "61964eee91e5892e990d564365d4e238"
server_key_version: "encryptionkey@0"
I20260620 22:01:11.863113 20413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:11.873956 20413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:11.875653 20413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:11.875803 20413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:11.879776 20413 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:33131
I20260620 22:01:11.879808 20480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:33131 every 8 connection(s)
I20260620 22:01:11.880141 20413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:01:11.880892 20481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:11.882236 20481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:01:11.882930 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 20413
I20260620 22:01:11.883024 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:01:11.883220 7110 external_mini_cluster.cc:1428] Setting key 1acaa8149ce54c436447344fb33255e1
Jun 20 22:01:11 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.241.189@KRBTEST.COM; defaulting to no policy
I20260620 22:01:11.891844 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:11.882836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.6.241.190:33131)
0620 22:01:11.883005 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:11.883015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:11.883197 (+ 182us) server_negotiation.cc:407] Connection header received
0620 22:01:11.883765 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:11.883767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:11.883786 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:11.883886 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:11.884392 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:11.885206 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:11.886012 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:11.886529 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:11.886677 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:11.888629 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:11.888637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:11.888637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:11.888645 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:11.890454 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:11.890898 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:11.890901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:11.890901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:11.890995 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:11.891184 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:11.891185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:11.891185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:11.891358 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:11.891413 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:11.891578 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 22:01:11.891668 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260620 22:01:11.891848 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:11.882833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33131 (local address 127.0.0.1:52250)
0620 22:01:11.883092 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:11.883126 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:11.883619 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:11.883889 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:11.883891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:11.884160 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:11.884296 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:11.884300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:11.885291 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:11.885292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:11.885930 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:11.885931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:11.886654 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:11.886655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:11.886723 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:11.887213 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:11.887221 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:11.888541 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:11.890543 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:11.890549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:11.890550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:11.890820 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:11.891069 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:11.891070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:11.891070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:11.891144 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:11.891410 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:11.891414 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:11.891464 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 22:01:11.891532 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:01:11.891738 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":44,"thread_start_us":86,"threads_started":1}
I20260620 22:01:11.892593 20481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:01:11.892920 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:11.892794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:39675 (local address 127.0.0.1:39214)
0620 22:01:11.892851 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:11.892896 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.189@KRBTEST.COM" created.
W20260620 22:01:11.893230 20429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
I20260620 22:01:11.893630 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:11.893526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:39675 (local address 127.0.0.1:39224)
0620 22:01:11.893572 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:11.893599 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:01:11.893749 20481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.189:39675: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab.
Entry for principal kudu/127.6.241.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab.
I20260620 22:01:11.906548 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.189:39675
--webserver_interface=127.6.241.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab
--principal=kudu/127.6.241.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:33131,127.6.241.189:39675,127.6.241.188:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:11.942951 20481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } attempt: 1
I20260620 22:01:11.943393 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:11.943274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:39675 (local address 127.0.0.1:39232)
0620 22:01:11.943334 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:11.943367 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 22:01:11.943781 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:11.943647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:39675 (local address 127.0.0.1:39244)
0620 22:01:11.943689 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:11.943744 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:01:11.943871 20481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.189:39675: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
W20260620 22:01:11.983880 20488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:11.984059 20488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:11.984081 20488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:11.985196 20488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:11.985234 20488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:11.985248 20488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:11.985260 20488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:11.985312 20488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:11.986773 20488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:33131,127.6.241.189:39675,127.6.241.188:37829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab
--principal=kudu/127.6.241.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.189:39675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.241.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:11.986971 20488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:11.987175 20488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:11.991708 20494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:11.991699 20493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:11.991699 20496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:11.991905 20488 server_base.cc:1034] running on GCE node
I20260620 22:01:11.992130 20488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:11.992421 20488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:11.993575 20488 hybrid_clock.cc:648] HybridClock initialized: now 1781992871993570 us; error 33 us; skew 500 ppm
Jun 20 22:01:11 dist-test-slave-gl2g krb5kdc[20400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:11.996049 20488 init.cc:377] Logged in from keytab as kudu/127.6.241.189@KRBTEST.COM (short username kudu)
I20260620 22:01:11.996459 20488 webserver.cc:460] Webserver started at http://127.6.241.189:36011/ using document root <none> and password file <none>
I20260620 22:01:11.996691 20488 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:11.996757 20488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:11.996907 20488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:11.997903 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/data/instance:
uuid: "aff969a28ffd4a5bae1eefc3f9c8b612"
format_stamp: "Formatted at 2026-06-20 22:01:11 on dist-test-slave-gl2g"
server_key: "462bebf8d6a8dbbd6c298fb47051d2fd"
server_key_iv: "d33ea99e1788bdd1804cc4337dcc524b"
server_key_version: "encryptionkey@0"
I20260620 22:01:11.998225 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/wal/instance:
uuid: "aff969a28ffd4a5bae1eefc3f9c8b612"
format_stamp: "Formatted at 2026-06-20 22:01:11 on dist-test-slave-gl2g"
server_key: "462bebf8d6a8dbbd6c298fb47051d2fd"
server_key_iv: "d33ea99e1788bdd1804cc4337dcc524b"
server_key_version: "encryptionkey@0"
I20260620 22:01:11.999490 20488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:01:12.000151 20503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:12.000342 20488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:01:12.000420 20488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/wal
uuid: "aff969a28ffd4a5bae1eefc3f9c8b612"
format_stamp: "Formatted at 2026-06-20 22:01:11 on dist-test-slave-gl2g"
server_key: "462bebf8d6a8dbbd6c298fb47051d2fd"
server_key_iv: "d33ea99e1788bdd1804cc4337dcc524b"
server_key_version: "encryptionkey@0"
I20260620 22:01:12.000473 20488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:12.012001 20481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } attempt: 2
I20260620 22:01:12.012370 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.012249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:39675 (local address 127.0.0.1:39258)
0620 22:01:12.012309 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.012344 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 22:01:12.012673 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.012579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:39675 (local address 127.0.0.1:39264)
0620 22:01:12.012624 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.012654 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:01:12.012766 20481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.189:39675: Network error: Client connection negotiation failed: client connection to 127.6.241.189:39675: connect: Connection refused (error 111)
I20260620 22:01:12.032843 20488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:12.034534 20488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:12.034708 20488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:12.038363 20488 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.189:39675
I20260620 22:01:12.038427 20555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.189:39675 every 8 connection(s)
I20260620 22:01:12.038770 20488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
I20260620 22:01:12.039631 20556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:12.040910 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 20488
I20260620 22:01:12.041025 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-1/wal/instance
I20260620 22:01:12.041225 7110 external_mini_cluster.cc:1428] Setting key 6c01c1d2fc82f1974603a59e5a7bf8d7
I20260620 22:01:12.041112 20556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.241.188@KRBTEST.COM; defaulting to no policy
I20260620 22:01:12.050709 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.041826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.6.241.190:33131)
0620 22:01:12.041890 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.041892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.042132 (+ 240us) server_negotiation.cc:407] Connection header received
0620 22:01:12.042815 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.042817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.042832 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.042907 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.043469 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.044004 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.045513 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.045708 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.045761 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.047980 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.047982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.047983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.047990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.049405 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.049784 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.049785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.049785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.049861 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.050092 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.050093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.050093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.050212 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.050263 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.050480 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.050592 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:01:12.050810 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.041802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33131 (local address 127.0.0.1:52260)
0620 22:01:12.041990 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.042035 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:12.042714 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:12.042956 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:12.042959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:12.043206 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:12.043379 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.043382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.044102 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.044103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.045428 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.045431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.045792 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.045794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.045949 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.046405 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:12.046416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:12.047886 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:12.049494 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.049500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.049502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.049708 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.049928 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.049929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.049929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.050026 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.050269 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:12.050273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:12.050375 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 22:01:12.050454 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 22:01:12.050573 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260620 22:01:12.051344 20556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.188@KRBTEST.COM" created.
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
I20260620 22:01:12.059794 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.051592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:39675 (local address 127.0.0.1:39280)
0620 22:01:12.051642 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.051653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:12.051694 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:12.052847 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:12.052848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:12.053005 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:12.053076 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.053078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.053873 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.053874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.054785 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.054788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.055216 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.055217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.055317 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.055664 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:12.055669 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:12.056923 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:12.058841 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.058842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.058842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.059055 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.059224 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.059225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.059225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.059274 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.059475 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:12.059475 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:12.059506 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 22:01:12.059558 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:01:12.059660 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:01:12.059820 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.051628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.6.241.189:39675)
0620 22:01:12.052710 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.052714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.052730 (+ 16us) server_negotiation.cc:407] Connection header received
0620 22:01:12.052749 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.052751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.052761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.052822 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.053149 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.053788 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.054859 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.055137 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.055191 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.057005 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.057013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.057013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.057020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.058756 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.059117 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.059120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.059120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.059171 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.059312 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.059312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.059313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.059427 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.059489 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.059618 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.059717 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":1058,"thread_start_us":82,"threads_started":1}
I20260620 22:01:12.060472 20556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:01:12.060813 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.060691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52242)
0620 22:01:12.060738 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.060786 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:01:12.060950 20507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
I20260620 22:01:12.061353 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.061272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52246)
0620 22:01:12.061313 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.061337 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:01:12.061445 20556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:37829: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab.
Entry for principal kudu/127.6.241.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab.
I20260620 22:01:12.065591 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.188:37829
--webserver_interface=127.6.241.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab
--principal=kudu/127.6.241.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:33131,127.6.241.189:39675,127.6.241.188:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:12.103899 20481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } attempt: 3
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
I20260620 22:01:12.110576 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.104227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:39675 (local address 127.0.0.1:39292)
0620 22:01:12.104279 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.104289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:12.104353 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:12.104501 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:12.104502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:12.104665 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:12.104728 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.104730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.105429 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.105430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.105939 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.105941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.106327 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.106328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.106406 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.106886 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:12.106901 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:12.108003 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:12.109709 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.109710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.109711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.109866 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.110057 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.110057 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.110057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.110110 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.110354 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:12.110355 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:12.110391 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 22:01:12.110440 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:01:12.110508 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:01:12.110671 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.104280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.6.241.189:39675)
0620 22:01:12.104342 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.104344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.104355 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:01:12.104412 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.104413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.104424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.104494 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.104833 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.105340 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.106024 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.106243 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.106326 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.108094 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.108096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.108096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.108122 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.109614 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.109933 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.109935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.109935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.109991 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.110173 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.110174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.110174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.110299 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.110359 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.110463 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.110537 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:01:12.110669 20556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } attempt: 1
I20260620 22:01:12.111081 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.110917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52260)
0620 22:01:12.111010 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.111054 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 22:01:12.111099 20481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:01:12.111455 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.111369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52276)
0620 22:01:12.111402 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.111432 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 22:01:12.111481 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.111403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52288)
0620 22:01:12.111435 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.111460 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 22:01:12.111732 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.111663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52296)
0620 22:01:12.111696 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.111714 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 22:01:12.111754 20556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:37829: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
W20260620 22:01:12.111812 20481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:37829: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
I20260620 22:01:12.142946 20481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } attempt: 1
I20260620 22:01:12.143319 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.143181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52300)
0620 22:01:12.143241 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.143288 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:01:12.143671 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.143583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52312)
0620 22:01:12.143623 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.143651 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 22:01:12.143769 20481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:37829: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
W20260620 22:01:12.147559 20563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:12.147746 20563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:12.147768 20563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:12.148903 20563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:12.148939 20563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:12.148954 20563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:12.149006 20563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:12.149029 20563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:12.150482 20563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:33131,127.6.241.189:39675,127.6.241.188:37829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab
--principal=kudu/127.6.241.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.188:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.241.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:12.150766 20563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:12.150978 20563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:01:12.155360 20563 server_base.cc:1034] running on GCE node
W20260620 22:01:12.155335 20569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:12.155335 20571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:12.155339 20568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:12.155735 20563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:12.155953 20563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:12.157086 20563 hybrid_clock.cc:648] HybridClock initialized: now 1781992872157065 us; error 35 us; skew 500 ppm
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:12.162405 20563 init.cc:377] Logged in from keytab as kudu/127.6.241.188@KRBTEST.COM (short username kudu)
I20260620 22:01:12.162863 20563 webserver.cc:460] Webserver started at http://127.6.241.188:44291/ using document root <none> and password file <none>
I20260620 22:01:12.163074 20563 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:12.163139 20563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:12.163286 20563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:12.164259 20563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/data/instance:
uuid: "5a43011e8b1d4d9a8850be92bddb0304"
format_stamp: "Formatted at 2026-06-20 22:01:12 on dist-test-slave-gl2g"
server_key: "a3f1de68367fd9c582fb0f3904c0b121"
server_key_iv: "053ec8fea9d1926224d8a1e6c69f417e"
server_key_version: "encryptionkey@0"
I20260620 22:01:12.164608 20563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/wal/instance:
uuid: "5a43011e8b1d4d9a8850be92bddb0304"
format_stamp: "Formatted at 2026-06-20 22:01:12 on dist-test-slave-gl2g"
server_key: "a3f1de68367fd9c582fb0f3904c0b121"
server_key_iv: "053ec8fea9d1926224d8a1e6c69f417e"
server_key_version: "encryptionkey@0"
I20260620 22:01:12.165867 20563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:01:12.166553 20578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:12.166724 20563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:01:12.166816 20563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/wal
uuid: "5a43011e8b1d4d9a8850be92bddb0304"
format_stamp: "Formatted at 2026-06-20 22:01:12 on dist-test-slave-gl2g"
server_key: "a3f1de68367fd9c582fb0f3904c0b121"
server_key_iv: "053ec8fea9d1926224d8a1e6c69f417e"
server_key_version: "encryptionkey@0"
I20260620 22:01:12.166899 20563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:12.179451 20563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:12.179893 20556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } attempt: 2
I20260620 22:01:12.180222 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.180113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52320)
0620 22:01:12.180164 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.180198 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:01:12.180756 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.180663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52324)
0620 22:01:12.180707 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.180732 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:01:12.181056 20556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:37829: Network error: Client connection negotiation failed: client connection to 127.6.241.188:37829: connect: Connection refused (error 111)
I20260620 22:01:12.181425 20563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:12.181572 20563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:12.185315 20563 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.188:37829
I20260620 22:01:12.185343 20630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.188:37829 every 8 connection(s)
I20260620 22:01:12.185657 20563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
I20260620 22:01:12.186391 20631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:12.187753 20631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:01:12.189831 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 20563
I20260620 22:01:12.189940 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992718478901-7110-0/minicluster-data/master-2/wal/instance
I20260620 22:01:12.190146 7110 external_mini_cluster.cc:1428] Setting key 89dbf4421c55f3efa8d125132eea9b0b
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:01:12.197588 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.188353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.6.241.190:33131)
0620 22:01:12.188403 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.188405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.188595 (+ 190us) server_negotiation.cc:407] Connection header received
0620 22:01:12.188981 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.188982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.188992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.189044 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.189573 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.190270 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.191609 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.192368 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.192443 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.194893 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.194894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.194894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.194901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.196187 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.196611 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.196612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.196613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.196675 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.196952 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.196952 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.196953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.197100 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.197170 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.197410 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.197478 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:01:12.197781 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.188334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33131 (local address 127.0.0.1:52270)
0620 22:01:12.188482 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.188513 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:12.188907 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:12.189089 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:12.189092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:12.189317 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:12.189505 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.189509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.190364 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.190365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.191506 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.191510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.192452 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.192452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.192560 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.192962 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:12.192972 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:12.194816 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:12.196286 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.196293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.196294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.196518 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.196750 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.196751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.196751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.196879 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.197175 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:12.197180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:12.197282 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 22:01:12.197364 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:01:12.197528 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260620 22:01:12.198290 20631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:01:12.199469 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.190502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.6.241.188:37829)
0620 22:01:12.190814 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.190818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.190830 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:01:12.190846 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.190847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.190862 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.190914 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.191608 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.192637 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.193675 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.194155 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.194220 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.196657 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.196662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.196662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.196667 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.198374 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.198771 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.198773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.198774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.198830 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.199002 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.199003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.199003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.199132 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.199255 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.199353 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.199404 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":115,"threads_started":1}
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:12.205345 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.198560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.6.241.189:39675)
0620 22:01:12.198635 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.198637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.198646 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:01:12.198679 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.198680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.198690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.198750 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.198999 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.199636 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.200477 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.200797 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.200872 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.202663 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.202665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.202665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.202672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.204233 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.204614 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.204615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.204616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.204674 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.204880 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.204880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.204880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.204975 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.205019 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.205186 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.205243 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 22:01:12.205346 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.198498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:39675 (local address 127.0.0.1:39298)
0620 22:01:12.198559 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.198567 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:12.198636 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:12.198759 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:12.198759 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:12.198899 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:12.198927 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.198928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.199701 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.199702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.200402 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.200405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.200869 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.200869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.200910 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.201257 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:12.201262 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:12.202548 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:12.204305 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.204306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.204307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.204520 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.204745 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.204746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.204746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.204810 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.205045 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:12.205046 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:12.205090 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 22:01:12.205163 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:01:12.205249 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 22:01:12.205789 20631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:01:12.207669 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.200430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.6.241.190:33131)
0620 22:01:12.200467 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.200469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.200478 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:01:12.200498 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.200499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.200506 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.200564 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.201126 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.201802 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.202694 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.202869 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.202942 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.205087 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.205088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.205089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.205095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.206582 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.206960 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.206960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.206961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.207005 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.207208 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.207208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.207209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.207319 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.207489 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.207523 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.207588 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
I20260620 22:01:12.212508 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.206047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.6.241.188:37829)
0620 22:01:12.206076 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.206078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.206085 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:01:12.206134 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.206135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.206143 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.206190 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.206409 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.207079 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.207945 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.208169 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.208248 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.209963 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.209964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.209964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.209970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.211413 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.211760 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.211761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.211761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.211808 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.211970 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.211971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.211971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.212075 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.212162 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.212284 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.212398 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":15}
I20260620 22:01:12.212674 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.205995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52330)
0620 22:01:12.206023 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.206033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:12.206075 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:12.206192 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:12.206193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:12.206317 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:12.206337 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.206338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.207148 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.207148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.207866 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.207868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.208405 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.208405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.208439 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.208816 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:12.208820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:12.209877 (+ 1057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:12.211510 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.211511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.211512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.211681 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.211857 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.211858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.211858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.211906 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.212156 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:12.212156 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:12.212192 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 22:01:12.212259 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 22:01:12.212591 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260620 22:01:12.214354 20631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304: Bootstrap starting.
I20260620 22:01:12.214761 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.208260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.6.241.189:39675)
0620 22:01:12.208295 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.208296 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.208303 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:01:12.208320 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.208320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.208328 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.208374 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.208768 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.209293 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.209994 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.210197 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.210268 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.212194 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.212196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.212196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.212203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.213618 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.213917 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.213918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.213919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.213975 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.214121 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.214122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.214122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.214240 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.214425 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.214464 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.214667 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:01:12.215317 20631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:12.215631 20631 log.cc:824] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:12.216310 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:01:12.216421 20631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304: No bootstrap required, opened a new log
I20260620 22:01:12.218109 20631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } }
I20260620 22:01:12.218230 20631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:12.218250 20631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a43011e8b1d4d9a8850be92bddb0304, State: Initialized, Role: FOLLOWER
I20260620 22:01:12.218358 20631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } }
I20260620 22:01:12.218659 20631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:12.218947 20481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } attempt: 2
I20260620 22:01:12.218649 20644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.220494 20644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:01:12.221163 20656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:01:12.221261 20656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:01:12.220855 20563 master_runner.cc:186] Error getting master registration for 127.6.241.190:33131: OK, instance_id { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" instance_seqno: 1781992871878546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:01:12.223452 20563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:01:12.224920 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.217615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.6.241.190:33131)
0620 22:01:12.217683 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.217685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.217821 (+ 136us) server_negotiation.cc:407] Connection header received
0620 22:01:12.217932 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.217933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.217943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.218006 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.218667 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.219297 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.220272 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.220540 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.220610 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.222085 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.222086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.222087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.222093 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.223688 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.224042 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.224043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.224043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.224095 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.224336 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.224336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.224337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.224451 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.224532 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.224737 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.224827 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:01:12.228669 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.217798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.6.241.188:37829)
0620 22:01:12.217851 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.217852 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.218943 (+ 1091us) server_negotiation.cc:407] Connection header received
0620 22:01:12.219026 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.219027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.219044 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.219071 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.222651 (+ 3580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.223273 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.223999 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.224295 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.224352 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.225856 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.225857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.225857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.225864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.227467 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.227812 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.227813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.227813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.227865 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.228068 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.228069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.228069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.228210 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.228277 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.228466 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.228592 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:01:12.229975 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.217698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.6.241.189:39675)
0620 22:01:12.217731 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.217733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.218780 (+ 1047us) server_negotiation.cc:407] Connection header received
0620 22:01:12.218849 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.218850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.218872 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.218921 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.220941 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.221645 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.224043 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.224362 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.224400 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.226672 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.226673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.226673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.226679 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.228273 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.228712 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.228713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.228713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.228765 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.228951 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.228952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.228952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.229062 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.229592 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.229818 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.229891 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:01:12.231997 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.219197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52348)
0620 22:01:12.219282 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.219292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:12.219327 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:12.222326 (+ 2999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:12.222327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:12.222471 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:12.222508 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.222510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.225695 (+ 3185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.225696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.226521 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.226524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.227053 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.227054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.227108 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.227435 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:12.227441 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:12.228828 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:12.230980 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.230980 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.230981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.231142 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.231371 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.231371 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.231372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.231426 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.231672 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:12.231673 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:12.231712 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 22:01:12.231820 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 22:01:12.231904 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 22:01:12.232010 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.219274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.6.241.188:37829)
0620 22:01:12.221009 (+ 1735us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.221011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.221018 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:01:12.221035 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.221036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.221045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.222899 (+ 1854us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.222960 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.225610 (+ 2650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.226629 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.226936 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.227052 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.229157 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.229158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.229158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.229163 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.230890 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.231226 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.231227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.231227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.231302 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.231492 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.231493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.231494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.231601 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.231668 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.231847 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.231913 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1692,"thread_start_us":82,"threads_started":1}
I20260620 22:01:12.233155 20481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920: Bootstrap starting.
I20260620 22:01:12.233806 20481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:12.234073 20481 log.cc:824] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:12.234817 20481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920: No bootstrap required, opened a new log
I20260620 22:01:12.236270 20481 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } }
I20260620 22:01:12.236366 20481 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:12.236400 20481 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3fe1c4b8ee40dfa80451360cc9d920, State: Initialized, Role: FOLLOWER
I20260620 22:01:12.236560 20481 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } }
I20260620 22:01:12.236833 20481 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:12.236821 20659 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.236932 20659 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:01:12.238526 20670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:01:12.238648 20670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:01:12.239235 20413 master_runner.cc:186] Error getting master registration for 127.6.241.189:39675: OK, instance_id { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" instance_seqno: 1781992872037345 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:01:12.241849 20413 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:01:12.272212 20556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } attempt: 3
Jun 20 22:01:12 dist-test-slave-gl2g krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:01:12.284941 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.272623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:37829 (local address 127.0.0.1:52362)
0620 22:01:12.272677 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:12.272689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:12.272727 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:12.272893 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:12.272894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:12.273087 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:12.273139 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.273142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.274117 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.274119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.274961 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:12.274964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.275472 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.275473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:12.275533 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.280239 (+ 4706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:12.280244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:12.281869 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:12.283796 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.283797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.283798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.284025 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.284262 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:12.284263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:12.284263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:12.284334 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:12.284633 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:12.284634 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:12.284681 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 22:01:12.284751 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:01:12.284843 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 22:01:12.284962 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:12.272706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.6.241.188:37829)
0620 22:01:12.272749 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:12.272751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:12.272761 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:01:12.272784 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:12.272785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:12.272803 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:12.272873 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:12.273221 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.274029 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.275047 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:12.275382 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:12.275465 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:12.281955 (+ 6490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:12.281956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:12.281956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:12.281963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:12.283703 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.284107 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.284109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.284109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.284175 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:12.284417 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:12.284418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:12.284419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:12.284553 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:12.284619 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:12.284789 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 22:01:12.284873 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:01:12.286972 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612: Bootstrap starting.
I20260620 22:01:12.287719 20556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:12.288020 20556 log.cc:824] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:12.289018 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612: No bootstrap required, opened a new log
I20260620 22:01:12.291052 20556 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } }
I20260620 22:01:12.291190 20556 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:12.291226 20556 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff969a28ffd4a5bae1eefc3f9c8b612, State: Initialized, Role: FOLLOWER
I20260620 22:01:12.291402 20556 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } }
I20260620 22:01:12.291723 20556 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:12.291692 20673 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.291903 20673 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:01:12.293511 20684 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:01:12.293612 20684 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:01:12.295519 20488 master_runner.cc:263] No leader master found from master aff969a28ffd4a5bae1eefc3f9c8b612
I20260620 22:01:12.295565 20488 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:01:12.308286 20659 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:01:12.308403 20659 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } }
I20260620 22:01:12.308605 20659 leader_election.cc:288] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff969a28ffd4a5bae1eefc3f9c8b612 (127.6.241.189:39675), 5a43011e8b1d4d9a8850be92bddb0304 (127.6.241.188:37829)
I20260620 22:01:12.309017 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a43011e8b1d4d9a8850be92bddb0304" is_pre_election: true
I20260620 22:01:12.309361 20605 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee3fe1c4b8ee40dfa80451360cc9d920 in term 0.
I20260620 22:01:12.309679 20432 leader_election.cc:302] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a43011e8b1d4d9a8850be92bddb0304, ee3fe1c4b8ee40dfa80451360cc9d920; no voters:
I20260620 22:01:12.310235 20530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" is_pre_election: true
I20260620 22:01:12.310405 20530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee3fe1c4b8ee40dfa80451360cc9d920 in term 0.
I20260620 22:01:12.310937 20659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:01:12.310998 20659 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:01:12.311022 20659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:12.312103 20659 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } }
I20260620 22:01:12.312220 20659 leader_election.cc:288] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [CANDIDATE]: Term 1 election: Requested vote from peers aff969a28ffd4a5bae1eefc3f9c8b612 (127.6.241.189:39675), 5a43011e8b1d4d9a8850be92bddb0304 (127.6.241.188:37829)
I20260620 22:01:12.312583 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a43011e8b1d4d9a8850be92bddb0304"
I20260620 22:01:12.312656 20605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:12.313571 20605 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee3fe1c4b8ee40dfa80451360cc9d920 in term 1.
I20260620 22:01:12.313735 20432 leader_election.cc:302] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a43011e8b1d4d9a8850be92bddb0304, ee3fe1c4b8ee40dfa80451360cc9d920; no voters:
I20260620 22:01:12.313897 20530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612"
I20260620 22:01:12.313966 20530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:12.314837 20530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee3fe1c4b8ee40dfa80451360cc9d920 in term 1.
I20260620 22:01:12.315095 20659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:12.315233 20659 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [term 1 LEADER]: Becoming Leader. State: Replica: ee3fe1c4b8ee40dfa80451360cc9d920, State: Running, Role: LEADER
I20260620 22:01:12.315313 20659 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } }
I20260620 22:01:12.315697 20659 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee3fe1c4b8ee40dfa80451360cc9d920. Latest consensus state: current_term: 1 leader_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.315763 20659 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:12.316087 20688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:01:12.316303 20688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:01:12.319324 20605 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [term 1 FOLLOWER]: Refusing update from remote peer ee3fe1c4b8ee40dfa80451360cc9d920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:12.319671 20659 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:12.321153 20644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee3fe1c4b8ee40dfa80451360cc9d920. Latest consensus state: current_term: 1 leader_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.321234 20644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:01:12.321642 20530 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [term 1 FOLLOWER]: Refusing update from remote peer ee3fe1c4b8ee40dfa80451360cc9d920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:12.321894 20659 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:12.322996 20673 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee3fe1c4b8ee40dfa80451360cc9d920. Latest consensus state: current_term: 1 leader_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.323081 20673 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:01:12.323804 20686 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.323887 20686 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:12.324384 20673 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.324466 20673 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:01:12.324579 20689 mvcc.cc:205] Tried to move back new op lower bound from 7299042805017874432 to 7299042805004070912. Current Snapshot: MvccSnapshot[applied={T|T < 7299042805017874432}]
I20260620 22:01:12.325469 20692 mvcc.cc:205] Tried to move back new op lower bound from 7299042805017874432 to 7299042805004070912. Current Snapshot: MvccSnapshot[applied={T|T < 7299042805017874432}]
I20260620 22:01:12.325769 20659 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.325836 20659 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:12.326210 20644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3fe1c4b8ee40dfa80451360cc9d920" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33131 } } peers { permanent_uuid: "aff969a28ffd4a5bae1eefc3f9c8b612" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 39675 } } peers { permanent_uuid: "5a43011e8b1d4d9a8850be92bddb0304" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 37829 } } }
I20260620 22:01:12.326280 20644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:01:12.326958 20690 mvcc.cc:205] Tried to move back new op lower bound from 7299042805017874432 to 7299042805004070912. Current Snapshot: MvccSnapshot[applied={T|T < 7299042805017874432}]
I20260620 22:01:12.327467 20688 catalog_manager.cc:1347] Generated new cluster ID: 1ad07c81b9134c379c88cf67117e60f7
I20260620 22:01:12.327507 20688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:01:12.361867 20688 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:01:12.362779 20688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:01:12.371795 20688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee3fe1c4b8ee40dfa80451360cc9d920: Generated new TSK 0
I20260620 22:01:12.372118 20688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:01:13.221748 20656 catalog_manager.cc:1259] Loaded cluster ID: 1ad07c81b9134c379c88cf67117e60f7
I20260620 22:01:13.221848 20656 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304: loading cluster ID for follower catalog manager: success
I20260620 22:01:13.222797 20656 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304: acquiring CA information for follower catalog manager: success
I20260620 22:01:13.223127 20656 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a43011e8b1d4d9a8850be92bddb0304: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:01:13.294061 20684 catalog_manager.cc:1259] Loaded cluster ID: 1ad07c81b9134c379c88cf67117e60f7
I20260620 22:01:13.294142 20684 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612: loading cluster ID for follower catalog manager: success
I20260620 22:01:13.295116 20684 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612: acquiring CA information for follower catalog manager: success
I20260620 22:01:13.295506 20684 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aff969a28ffd4a5bae1eefc3f9c8b612: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:01:13.449026 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 20413
I20260620 22:01:13.453809 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 20488
I20260620 22:01:13.458379 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 20563
2026-06-20T22:01:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2818 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:01:13 dist-test-slave-gl2g krb5kdc[20703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:01:13 dist-test-slave-gl2g krb5kdc[20703](info): set up 2 sockets
Jun 20 22:01:13 dist-test-slave-gl2g krb5kdc[20703](info): commencing operation
krb5kdc: starting...
W20260620 22:01:14.498799 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:01:14 dist-test-slave-gl2g krb5kdc[20703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:01:14.622211 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34373
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:14.704512 20716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:14.704772 20716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:14.704811 20716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:14.705979 20716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:14.706053 20716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:14.706077 20716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:14.706106 20716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:14.706132 20716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:14.707779 20716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:14.708081 20716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:14.708320 20716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:14.713196 20721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:14.713187 20724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:14.713232 20716 server_base.cc:1034] running on GCE node
W20260620 22:01:14.713187 20722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:14.713635 20716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:14.713964 20716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:14.715126 20716 hybrid_clock.cc:648] HybridClock initialized: now 1781992874715105 us; error 38 us; skew 500 ppm
Jun 20 22:01:14 dist-test-slave-gl2g krb5kdc[20703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992874, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:14.717509 20716 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:01:14.717934 20716 webserver.cc:460] Webserver started at http://127.6.241.190:44151/ using document root <none> and password file <none>
I20260620 22:01:14.718175 20716 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:14.718246 20716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:14.718395 20716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:14.719522 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "021b3226ab8c425ea8e66e5dcfa4587d"
format_stamp: "Formatted at 2026-06-20 22:01:14 on dist-test-slave-gl2g"
server_key: "9259e8c16deb553bed871d9314a9b526"
server_key_iv: "1f4638000fc25faf666747c1ac67962b"
server_key_version: "encryptionkey@0"
I20260620 22:01:14.719882 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "021b3226ab8c425ea8e66e5dcfa4587d"
format_stamp: "Formatted at 2026-06-20 22:01:14 on dist-test-slave-gl2g"
server_key: "9259e8c16deb553bed871d9314a9b526"
server_key_iv: "1f4638000fc25faf666747c1ac67962b"
server_key_version: "encryptionkey@0"
I20260620 22:01:14.721205 20716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:01:14.721935 20731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:14.722114 20716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:01:14.722204 20716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "021b3226ab8c425ea8e66e5dcfa4587d"
format_stamp: "Formatted at 2026-06-20 22:01:14 on dist-test-slave-gl2g"
server_key: "9259e8c16deb553bed871d9314a9b526"
server_key_iv: "1f4638000fc25faf666747c1ac67962b"
server_key_version: "encryptionkey@0"
I20260620 22:01:14.722286 20716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:14.732832 20716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:14.734711 20716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:14.734869 20716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:14.738780 20716 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:34373
I20260620 22:01:14.738811 20783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:34373 every 8 connection(s)
I20260620 22:01:14.739176 20716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:01:14.740062 20784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:14.742327 20784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d: Bootstrap starting.
I20260620 22:01:14.742969 20784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:14.743260 20784 log.cc:824] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:14.743948 20784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d: No bootstrap required, opened a new log
I20260620 22:01:14.745500 20784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021b3226ab8c425ea8e66e5dcfa4587d" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34373 } }
I20260620 22:01:14.745637 20784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:14.745685 20784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021b3226ab8c425ea8e66e5dcfa4587d, State: Initialized, Role: FOLLOWER
I20260620 22:01:14.745836 20784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021b3226ab8c425ea8e66e5dcfa4587d" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34373 } }
I20260620 22:01:14.745898 20784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:01:14.745939 20784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:01:14.745990 20784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:14.746501 20784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021b3226ab8c425ea8e66e5dcfa4587d" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34373 } }
I20260620 22:01:14.746611 20784 leader_election.cc:302] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 021b3226ab8c425ea8e66e5dcfa4587d; no voters:
I20260620 22:01:14.746830 20784 leader_election.cc:288] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:01:14.746892 20787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:14.747069 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 20716
I20260620 22:01:14.747080 20787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [term 1 LEADER]: Becoming Leader. State: Replica: 021b3226ab8c425ea8e66e5dcfa4587d, State: Running, Role: LEADER
I20260620 22:01:14.747103 20784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:14.747162 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:01:14.747164 20787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021b3226ab8c425ea8e66e5dcfa4587d" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34373 } }
I20260620 22:01:14.747372 7110 external_mini_cluster.cc:1428] Setting key b873c2eb47c17f11c7ad37b93e839f0c
I20260620 22:01:14.747792 20788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "021b3226ab8c425ea8e66e5dcfa4587d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021b3226ab8c425ea8e66e5dcfa4587d" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34373 } } }
I20260620 22:01:14.747869 20788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:14.748204 20797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:01:14.748205 20789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 021b3226ab8c425ea8e66e5dcfa4587d. Latest consensus state: current_term: 1 leader_uuid: "021b3226ab8c425ea8e66e5dcfa4587d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021b3226ab8c425ea8e66e5dcfa4587d" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34373 } } }
I20260620 22:01:14.748412 20789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:14.748979 20797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:01:14.750787 20797 catalog_manager.cc:1347] Generated new cluster ID: d68d9c6d57fc43f98737b1a4466a5c0b
I20260620 22:01:14.750844 20797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:01:14 dist-test-slave-gl2g krb5kdc[20703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:14.755330 20797 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:01:14.755926 20797 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:01:14.759384 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:14.747999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.6.241.190:34373)
0620 22:01:14.748432 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:14.748442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:14.748479 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:01:14.748604 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:14.748613 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:14.748684 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:14.748832 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:14.749643 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:14.751031 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:14.752122 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:14.752758 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:14.752879 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:14.755775 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:14.755779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:14.755780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:14.755792 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:14.757740 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:14.758161 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:14.758164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:14.758164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:14.758264 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:14.758473 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:14.758474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:14.758474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:14.758755 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:14.758902 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:14.759017 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 22:01:14.759160 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":67,"threads_started":1}
I20260620 22:01:14.760308 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:01:14.761715 20797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 021b3226ab8c425ea8e66e5dcfa4587d: Generated new TSK 0
I20260620 22:01:14.761848 20797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:01:14.814268 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 20716
2026-06-20T22:01:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1354 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:01:14 dist-test-slave-gl2g krb5kdc[20811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:01:14 dist-test-slave-gl2g krb5kdc[20811](info): set up 2 sockets
Jun 20 22:01:14 dist-test-slave-gl2g krb5kdc[20811](info): commencing operation
krb5kdc: starting...
W20260620 22:01:15.850807 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:01:15 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:01:15.969813 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:32893
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:16.054011 20824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:16.054224 20824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:16.054246 20824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:16.055476 20824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:16.055523 20824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:16.055538 20824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:16.055552 20824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:16.055613 20824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:16.057116 20824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:32893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:16.057377 20824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:16.057636 20824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:16.062419 20832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:16.062441 20830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:16.062397 20829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:16.062686 20824 server_base.cc:1034] running on GCE node
I20260620 22:01:16.062902 20824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:16.063207 20824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:16.064360 20824 hybrid_clock.cc:648] HybridClock initialized: now 1781992876064337 us; error 39 us; skew 500 ppm
Jun 20 22:01:16 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992876, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:16.066784 20824 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:01:16.067209 20824 webserver.cc:460] Webserver started at http://127.6.241.190:44175/ using document root <none> and password file <none>
I20260620 22:01:16.067430 20824 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:16.067476 20824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:16.067641 20824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:16.068693 20824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "f4c6a9a580864beba2d1584e798973de"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "e5e348323b33520dbde9fa4b2f7dc073"
server_key_iv: "f72986563477c02f5375e112214f4eb1"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.069047 20824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "f4c6a9a580864beba2d1584e798973de"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "e5e348323b33520dbde9fa4b2f7dc073"
server_key_iv: "f72986563477c02f5375e112214f4eb1"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.070338 20824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:01:16.071120 20839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.071305 20824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.071395 20824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f4c6a9a580864beba2d1584e798973de"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "e5e348323b33520dbde9fa4b2f7dc073"
server_key_iv: "f72986563477c02f5375e112214f4eb1"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.071473 20824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:16.084131 20824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:16.086134 20824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:16.086274 20824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:16.090335 20824 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:32893
I20260620 22:01:16.090399 20891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:32893 every 8 connection(s)
I20260620 22:01:16.090801 20824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:01:16.091768 20892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:16.094110 20892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Bootstrap starting.
I20260620 22:01:16.094774 20892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:16.094986 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 20824
I20260620 22:01:16.095077 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:01:16.095100 20892 log.cc:824] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:16.095304 7110 external_mini_cluster.cc:1428] Setting key cfc962181119782797c3d0610557ea59
I20260620 22:01:16.095939 20892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: No bootstrap required, opened a new log
I20260620 22:01:16.098773 20892 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:16.098945 20892 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:16.098990 20892 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c6a9a580864beba2d1584e798973de, State: Initialized, Role: FOLLOWER
I20260620 22:01:16.099128 20892 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:16.099246 20892 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:01:16.099292 20892 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:01:16.099413 20892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:16.100067 20892 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:16.100174 20892 leader_election.cc:302] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c6a9a580864beba2d1584e798973de; no voters:
I20260620 22:01:16.100342 20892 leader_election.cc:288] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:01:16.100399 20897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:16.100596 20897 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 1 LEADER]: Becoming Leader. State: Replica: f4c6a9a580864beba2d1584e798973de, State: Running, Role: LEADER
I20260620 22:01:16.100605 20892 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:16.100744 20897 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:16.101127 20898 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4c6a9a580864beba2d1584e798973de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } } }
I20260620 22:01:16.101125 20899 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c6a9a580864beba2d1584e798973de. Latest consensus state: current_term: 1 leader_uuid: "f4c6a9a580864beba2d1584e798973de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } } }
I20260620 22:01:16.101230 20898 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:16.101234 20899 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:16.101919 20904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:01:16.102507 20904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:01:16 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:16.104210 20904 catalog_manager.cc:1347] Generated new cluster ID: a1a5f58dc47b4141bc25336e5ffc217a
I20260620 22:01:16.104279 20904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:01:16.107800 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.095814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.6.241.190:32893)
0620 22:01:16.096036 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.096049 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.096090 (+ 41us) server_negotiation.cc:407] Connection header received
0620 22:01:16.096590 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.096599 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.096662 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.096818 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:16.097441 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.098919 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.099837 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.100500 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.100927 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.103858 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:16.103861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:16.103862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:16.103873 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:16.105921 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.106344 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.106347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.106347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.106446 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.106699 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.106700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.106701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.107038 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:16.107207 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.107323 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.107578 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":101,"threads_started":1}
I20260620 22:01:16.117190 20904 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:01:16.117683 20904 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:01:16.122784 20904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Generated new TSK 0
I20260620 22:01:16.122999 20904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 22:01:16.153756 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:32893
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:16.234931 20918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:16.235177 20918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:16.235217 20918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:16.236390 20918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:16.236486 20918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:16.236532 20918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:01:16.238005 20918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:16.238267 20918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:16.238502 20918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:16.243745 20926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:16.243739 20924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:16.243804 20923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:16.243961 20918 server_base.cc:1034] running on GCE node
I20260620 22:01:16.244294 20918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:16.244536 20918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:16.245712 20918 hybrid_clock.cc:648] HybridClock initialized: now 1781992876245698 us; error 28 us; skew 500 ppm
Jun 20 22:01:16 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992876, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:16.248344 20918 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:01:16.248788 20918 webserver.cc:460] Webserver started at http://127.6.241.129:38945/ using document root <none> and password file <none>
I20260620 22:01:16.249014 20918 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:16.249091 20918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:16.249239 20918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:16.250330 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "4f1f69a40b554ba8be22cffbd4df46df"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "4915a7973f60b5ee604cd7004cb54a44"
server_key_iv: "9d5be450a1a58f7b463778f1b56aae2b"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.250753 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "4f1f69a40b554ba8be22cffbd4df46df"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "4915a7973f60b5ee604cd7004cb54a44"
server_key_iv: "9d5be450a1a58f7b463778f1b56aae2b"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.252087 20918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:01:16.252807 20933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.252981 20918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:01:16.253090 20918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "4f1f69a40b554ba8be22cffbd4df46df"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "4915a7973f60b5ee604cd7004cb54a44"
server_key_iv: "9d5be450a1a58f7b463778f1b56aae2b"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.253175 20918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:16.265261 20918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:16.267163 20918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:16.267320 20918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:16.267582 20918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:16.267908 20918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:16.267963 20918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.267998 20918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:16.268025 20918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.273801 20918 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:38673
I20260620 22:01:16.273872 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:38673 every 8 connection(s)
I20260620 22:01:16.274236 20918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:01:16.278769 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 20918
I20260620 22:01:16.278878 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:01:16.279124 7110 external_mini_cluster.cc:1428] Setting key 633f8dbd154a9fc44a66fd2a669f606e
Jun 20 22:01:16 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992876, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:16.283897 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.275052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:56637 (local address 127.6.241.190:32893)
0620 22:01:16.275116 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.275119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.275320 (+ 201us) server_negotiation.cc:407] Connection header received
0620 22:01:16.275777 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.275778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.275789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.275841 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:16.276763 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.277281 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.278337 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.278528 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.278704 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.281094 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:16.281095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:16.281096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:16.281104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:16.282477 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.282924 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.282925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.282926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.282987 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.283241 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.283242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.283242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.283355 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:16.283427 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.283712 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.283802 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:01:16.284106 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.275006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:56637)
0620 22:01:16.275193 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:16.275228 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:16.275696 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:16.275899 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:16.275902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:16.276491 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:16.276685 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.276690 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.277384 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.277385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.278256 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.278259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.278643 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.278644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.278794 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.279400 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:16.279410 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:16.280979 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:16.282563 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:16.282570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:16.282580 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:16.282830 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:16.283064 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:16.283064 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:16.283065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:16.283173 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:16.283428 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:16.283436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:16.283583 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 22:01:16.283692 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 22:01:16.283854 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":125,"threads_started":1}
I20260620 22:01:16.284585 21047 heartbeater.cc:347] Connected to a master server at 127.6.241.190:32893
I20260620 22:01:16.284696 21047 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:16.284906 21047 heartbeater.cc:511] Master 127.6.241.190:32893 requested a full tablet report, sending...
I20260620 22:01:16.285432 20856 ts_manager.cc:194] Registered new tserver with Master: 4f1f69a40b554ba8be22cffbd4df46df (127.6.241.129:38673)
I20260620 22:01:16.286049 20856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:56637
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 22:01:16.305959 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:32893
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:16.388461 21052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:16.388674 21052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:16.388700 21052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:16.389810 21052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:16.389846 21052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:16.389873 21052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:01:16.391366 21052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:16.391561 21052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:16.391736 21052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:16.396719 21060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:16.396683 21058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:16.396683 21057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:16.397007 21052 server_base.cc:1034] running on GCE node
I20260620 22:01:16.397294 21052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:16.397563 21052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:16.398679 21052 hybrid_clock.cc:648] HybridClock initialized: now 1781992876398649 us; error 37 us; skew 500 ppm
Jun 20 22:01:16 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992876, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:16.401463 21052 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:01:16.402045 21052 webserver.cc:460] Webserver started at http://127.6.241.130:34293/ using document root <none> and password file <none>
I20260620 22:01:16.402266 21052 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:16.402311 21052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:16.402413 21052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:16.403594 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "c692fd88b81549ce87ec0f34661d3b36"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "112c3dce80301f9856c569e5801636a4"
server_key_iv: "8dd8367b29bad024248e54132022c12c"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.403949 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "c692fd88b81549ce87ec0f34661d3b36"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "112c3dce80301f9856c569e5801636a4"
server_key_iv: "8dd8367b29bad024248e54132022c12c"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.405289 21052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:01:16.405983 21067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.406147 21052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:01:16.406260 21052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "c692fd88b81549ce87ec0f34661d3b36"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "112c3dce80301f9856c569e5801636a4"
server_key_iv: "8dd8367b29bad024248e54132022c12c"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.406339 21052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:16.428731 21052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:16.430974 21052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:16.431093 21052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:16.431270 21052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:16.431612 21052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:16.431672 21052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.431717 21052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:16.431746 21052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.438320 21052 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:35823
I20260620 22:01:16.438362 21180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:35823 every 8 connection(s)
I20260620 22:01:16.438776 21052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:01:16.441015 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 21052
I20260620 22:01:16.441100 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:01:16.441318 7110 external_mini_cluster.cc:1428] Setting key 3b0617e4aa1a35b27cef43cfaa3c1c8e
Jun 20 22:01:16 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992876, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:16.448659 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.439468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:36531 (local address 127.6.241.190:32893)
0620 22:01:16.439513 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.439515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.439742 (+ 227us) server_negotiation.cc:407] Connection header received
0620 22:01:16.440231 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.440232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.440243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.440310 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:16.441339 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.442076 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.443146 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.443425 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.443490 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.445741 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:16.445742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:16.445742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:16.445752 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:16.447256 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.447677 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.447678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.447678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.447742 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.447997 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.447998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.447998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.448115 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:16.448192 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.448478 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.448557 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:01:16.448880 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.439425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:36531)
0620 22:01:16.439610 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:16.439653 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:16.440139 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:16.440368 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:16.440372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:16.440974 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:16.441242 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.441249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.442163 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.442164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.443064 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.443067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.443516 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.443517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.443687 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.444253 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:16.444264 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:16.445632 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:16.447353 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:16.447360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:16.447362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:16.447598 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:16.447808 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:16.447809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:16.447810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:16.447923 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:16.448182 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:16.448188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:16.448335 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 22:01:16.448447 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 22:01:16.448636 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260620 22:01:16.449551 21181 heartbeater.cc:347] Connected to a master server at 127.6.241.190:32893
I20260620 22:01:16.449663 21181 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:16.449844 21181 heartbeater.cc:511] Master 127.6.241.190:32893 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
I20260620 22:01:16.450313 20856 ts_manager.cc:194] Registered new tserver with Master: c692fd88b81549ce87ec0f34661d3b36 (127.6.241.130:35823)
I20260620 22:01:16.450786 20856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:36531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 22:01:16.467438 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:32893
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:16.551474 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:16.551699 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:16.551723 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:16.552857 21186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:16.552896 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:16.552923 21186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:01:16.554324 21186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:16.554524 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:16.554775 21186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:16.559937 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:16.559934 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:16.559984 21186 server_base.cc:1034] running on GCE node
W20260620 22:01:16.559945 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:16.560398 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:16.560644 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:16.561794 21186 hybrid_clock.cc:648] HybridClock initialized: now 1781992876561780 us; error 31 us; skew 500 ppm
Jun 20 22:01:16 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992876, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:16.564517 21186 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:01:16.565024 21186 webserver.cc:460] Webserver started at http://127.6.241.131:35901/ using document root <none> and password file <none>
I20260620 22:01:16.565256 21186 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:16.565305 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:16.565474 21186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:16.566581 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "19345c3de51a40f7ac34406220433d8b"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "540cf96fb6216095f88cfdf49e33b15c"
server_key_iv: "fe7efaa812e25dcf464df2329ce2ffbd"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.567013 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "19345c3de51a40f7ac34406220433d8b"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "540cf96fb6216095f88cfdf49e33b15c"
server_key_iv: "fe7efaa812e25dcf464df2329ce2ffbd"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.568387 21186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:01:16.569147 21201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.569320 21186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:01:16.569427 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "19345c3de51a40f7ac34406220433d8b"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "540cf96fb6216095f88cfdf49e33b15c"
server_key_iv: "fe7efaa812e25dcf464df2329ce2ffbd"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.569504 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:16.581090 21186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:16.583624 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:16.583801 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:16.584043 21186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:16.584374 21186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:16.584431 21186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.584471 21186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:16.584501 21186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.590601 21186 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:44437
I20260620 22:01:16.590723 21314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:44437 every 8 connection(s)
I20260620 22:01:16.591033 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:01:16.592216 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 21186
I20260620 22:01:16.592296 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:01:16.592497 7110 external_mini_cluster.cc:1428] Setting key 7e26d3459c0b4abfd2a6d7deb4199b76
Jun 20 22:01:16 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992876, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:16.600502 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.591782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:37995 (local address 127.6.241.190:32893)
0620 22:01:16.591835 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.591838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.592076 (+ 238us) server_negotiation.cc:407] Connection header received
0620 22:01:16.592732 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.592733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.592743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.592839 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:16.594001 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.594499 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.595479 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.595680 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.595734 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.597909 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:16.597910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:16.597910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:16.597917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:16.599172 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.599523 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.599524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.599525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.599592 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.599891 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.599892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.599892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.599994 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:16.600043 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.600367 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.600432 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:01:16.600744 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.591768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:37995)
0620 22:01:16.591921 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:16.591958 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:16.592614 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:16.592932 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:16.592937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:16.593718 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:16.593921 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.593925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.594611 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.594612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.595407 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.595409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.595765 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.595766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.595893 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.596456 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:16.596465 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:16.597827 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:16.599239 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:16.599244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:16.599246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:16.599454 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:16.599688 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:16.599689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:16.599689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:16.599823 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:16.600066 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:16.600071 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:16.600250 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 22:01:16.600340 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 22:01:16.600474 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260620 22:01:16.601352 21315 heartbeater.cc:347] Connected to a master server at 127.6.241.190:32893
I20260620 22:01:16.601471 21315 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:16.601658 21315 heartbeater.cc:511] Master 127.6.241.190:32893 requested a full tablet report, sending...
I20260620 22:01:16.602041 20856 ts_manager.cc:194] Registered new tserver with Master: 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437)
I20260620 22:01:16.602504 20856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:37995
I20260620 22:01:16.604175 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:01:16.611819 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.605509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.6.241.190:32893)
0620 22:01:16.605600 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.605602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.605695 (+ 93us) server_negotiation.cc:407] Connection header received
0620 22:01:16.605761 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.605762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.605772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.605827 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:16.606387 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.606937 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.607715 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.607901 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.607983 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.609134 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:16.609136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:16.609136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:16.609143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:16.610689 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.610985 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.610986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.610986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.611035 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:16.611241 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:16.611242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:16.611242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:16.611402 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:16.611531 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.611606 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.611675 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 22:01:16.613399 20856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:01:16.615217 20856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:01:16.621841 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.617472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.6.241.130:35823)
0620 22:01:16.617649 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.617664 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.617877 (+ 213us) server_negotiation.cc:407] Connection header received
0620 22:01:16.617928 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.617932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.617948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.618029 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:16.618312 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.619523 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.620633 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.621381 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.621469 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.621534 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.621705 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.621751 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260620 22:01:16.621853 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.617599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:44437 (local address 127.0.0.1:49460)
0620 22:01:16.617912 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:16.617920 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:16.617953 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:16.618389 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:16.618392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:16.618394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:16.618443 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.618444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.619393 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.619394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.620661 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.620664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.621572 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.621573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.621628 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.621631 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:01:16.621701 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:01:16.621748 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":53,"threads_started":1}
I20260620 22:01:16.621860 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.617430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:38673 (local address 127.0.0.1:45272)
0620 22:01:16.617780 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:16.617789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:16.617839 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:16.618077 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:16.618077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:16.618079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:16.618250 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.618251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.619629 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.619630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.620865 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.620867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.621564 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.621565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.621624 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.621627 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:01:16.621656 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 22:01:16.621720 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":42,"thread_start_us":85,"threads_started":1}
I20260620 22:01:16.621860 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.617373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35823 (local address 127.0.0.1:60686)
0620 22:01:16.617670 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:16.617714 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:16.617789 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:16.618073 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:16.618076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:16.618079 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:16.618214 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.618216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.619646 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.619647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.620546 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.620548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.621484 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.621485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.621584 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.621602 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 22:01:16.621676 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 22:01:16.621720 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":29,"thread_start_us":77,"threads_started":1}
I20260620 22:01:16.622109 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.617782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.6.241.131:44437)
0620 22:01:16.618170 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.618175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.618186 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:01:16.618221 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.618223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.618238 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.618307 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:16.618522 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.619285 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.620754 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.621414 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.621484 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.621535 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.621965 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.622019 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":70,"threads_started":1}
I20260620 22:01:16.622673 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.617553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.6.241.129:38673)
0620 22:01:16.617765 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.617769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.617836 (+ 67us) server_negotiation.cc:407] Connection header received
0620 22:01:16.617908 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.617910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.617927 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.618326 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:16.618409 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.619519 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.620951 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.621468 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.622403 (+ 935us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.622479 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.622505 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.622551 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
I20260620 22:01:16.622885 21249 tablet_service.cc:1467] Processing CreateTablet for tablet 990df9ec657f458fa3e066aa0dd34c5d (DEFAULT_TABLE table=test-table [id=925f02f077e047fa8949f250c17a36c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:16.623317 21249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990df9ec657f458fa3e066aa0dd34c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:16.623368 20981 tablet_service.cc:1467] Processing CreateTablet for tablet 990df9ec657f458fa3e066aa0dd34c5d (DEFAULT_TABLE table=test-table [id=925f02f077e047fa8949f250c17a36c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:16.623684 20981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990df9ec657f458fa3e066aa0dd34c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:16.626925 21329 tablet_bootstrap.cc:492] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Bootstrap starting.
I20260620 22:01:16.626925 21330 tablet_bootstrap.cc:492] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: Bootstrap starting.
I20260620 22:01:16.627115 21115 tablet_service.cc:1467] Processing CreateTablet for tablet 990df9ec657f458fa3e066aa0dd34c5d (DEFAULT_TABLE table=test-table [id=925f02f077e047fa8949f250c17a36c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:16.627386 21115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990df9ec657f458fa3e066aa0dd34c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:16.627650 21329 tablet_bootstrap.cc:654] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:16.627665 21330 tablet_bootstrap.cc:654] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:16.628244 21329 log.cc:824] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:16.628244 21330 log.cc:824] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:16.629171 21330 tablet_bootstrap.cc:492] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: No bootstrap required, opened a new log
I20260620 22:01:16.629232 21330 ts_tablet_manager.cc:1397] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:01:16.629266 21329 tablet_bootstrap.cc:492] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: No bootstrap required, opened a new log
I20260620 22:01:16.629320 21329 ts_tablet_manager.cc:1397] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:01:16.630246 21333 tablet_bootstrap.cc:492] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: Bootstrap starting.
I20260620 22:01:16.630874 21333 tablet_bootstrap.cc:654] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:16.631211 21329 raft_consensus.cc:348] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:16.631211 21330 raft_consensus.cc:348] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:16.631351 21333 log.cc:824] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:16.631393 21329 raft_consensus.cc:374] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:16.631393 21330 raft_consensus.cc:374] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:16.631433 21329 raft_consensus.cc:729] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1f69a40b554ba8be22cffbd4df46df, State: Initialized, Role: FOLLOWER
I20260620 22:01:16.631433 21330 raft_consensus.cc:729] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19345c3de51a40f7ac34406220433d8b, State: Initialized, Role: FOLLOWER
I20260620 22:01:16.631554 21329 consensus_queue.cc:260] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:16.631589 21330 consensus_queue.cc:260] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:16.631856 21330 ts_tablet_manager.cc:1428] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:01:16.631856 21329 ts_tablet_manager.cc:1428] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:01:16.631999 21047 heartbeater.cc:503] Master 127.6.241.190:32893 was elected leader, sending a full tablet report...
I20260620 22:01:16.632112 21333 tablet_bootstrap.cc:492] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: No bootstrap required, opened a new log
I20260620 22:01:16.632176 21333 ts_tablet_manager.cc:1397] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:01:16.632310 21315 heartbeater.cc:503] Master 127.6.241.190:32893 was elected leader, sending a full tablet report...
I20260620 22:01:16.633929 21333 raft_consensus.cc:348] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:16.634068 21333 raft_consensus.cc:374] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:16.634088 21333 raft_consensus.cc:729] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c692fd88b81549ce87ec0f34661d3b36, State: Initialized, Role: FOLLOWER
I20260620 22:01:16.634209 21333 consensus_queue.cc:260] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:16.634472 21333 ts_tablet_manager.cc:1428] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:01:16.634519 21181 heartbeater.cc:503] Master 127.6.241.190:32893 was elected leader, sending a full tablet report...
W20260620 22:01:16.689386 21182 tablet.cc:2369] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:01:16.771406 21336 raft_consensus.cc:482] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:01:16.771545 21336 raft_consensus.cc:504] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:16.771958 21336 leader_election.cc:288] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c692fd88b81549ce87ec0f34661d3b36 (127.6.241.130:35823), 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437)
W20260620 22:01:16.774988 21048 tablet.cc:2369] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:01:16.776047 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.772272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:52081 (local address 127.6.241.130:35823)
0620 22:01:16.772346 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.772349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.772462 (+ 113us) server_negotiation.cc:407] Connection header received
0620 22:01:16.772577 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.772579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.772589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.772646 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:16.772858 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.773665 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.775242 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.775658 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.775756 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.775766 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.775932 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.775969 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:01:16.776082 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.772201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35823 (local address 127.6.241.129:52081)
0620 22:01:16.772363 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:16.772376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:16.772454 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:16.772660 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:16.772661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:16.772663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:16.772781 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.772783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.773732 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.773733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.775133 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.775137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.775754 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.775755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.775842 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.775850 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 22:01:16.775914 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 22:01:16.775963 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":100}
I20260620 22:01:16.776089 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.772427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:56575 (local address 127.6.241.131:44437)
0620 22:01:16.772467 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.772469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.772496 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:01:16.772577 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.772578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.772589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.772646 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:16.772858 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.773639 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.775278 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.775710 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.775805 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.775815 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.775978 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.776022 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:01:16.776217 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.772331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:44437 (local address 127.6.241.129:56575)
0620 22:01:16.772441 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:16.772451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:16.772506 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:16.772660 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:16.772661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:16.772663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:16.772781 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.772783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.773731 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.773732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.775133 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:16.775137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.775805 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.775807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:16.775898 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.775902 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:01:16.775971 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 22:01:16.776014 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":45,"thread_start_us":73,"threads_started":1}
I20260620 22:01:16.776424 21135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990df9ec657f458fa3e066aa0dd34c5d" candidate_uuid: "4f1f69a40b554ba8be22cffbd4df46df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c692fd88b81549ce87ec0f34661d3b36" is_pre_election: true
I20260620 22:01:16.776492 21269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990df9ec657f458fa3e066aa0dd34c5d" candidate_uuid: "4f1f69a40b554ba8be22cffbd4df46df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19345c3de51a40f7ac34406220433d8b" is_pre_election: true
I20260620 22:01:16.776608 21269 raft_consensus.cc:2454] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f1f69a40b554ba8be22cffbd4df46df in term 0.
I20260620 22:01:16.776608 21135 raft_consensus.cc:2454] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f1f69a40b554ba8be22cffbd4df46df in term 0.
I20260620 22:01:16.776819 20937 leader_election.cc:302] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19345c3de51a40f7ac34406220433d8b, 4f1f69a40b554ba8be22cffbd4df46df; no voters:
I20260620 22:01:16.776948 21336 raft_consensus.cc:2781] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:01:16.777005 21336 raft_consensus.cc:482] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:01:16.777032 21336 raft_consensus.cc:3037] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:16.777769 21336 raft_consensus.cc:504] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:16.777906 21336 leader_election.cc:288] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 1 election: Requested vote from peers c692fd88b81549ce87ec0f34661d3b36 (127.6.241.130:35823), 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437)
I20260620 22:01:16.778121 21269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990df9ec657f458fa3e066aa0dd34c5d" candidate_uuid: "4f1f69a40b554ba8be22cffbd4df46df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19345c3de51a40f7ac34406220433d8b"
I20260620 22:01:16.778121 21135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990df9ec657f458fa3e066aa0dd34c5d" candidate_uuid: "4f1f69a40b554ba8be22cffbd4df46df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c692fd88b81549ce87ec0f34661d3b36"
I20260620 22:01:16.778218 21269 raft_consensus.cc:3037] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:16.778215 21135 raft_consensus.cc:3037] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:16.778997 21269 raft_consensus.cc:2454] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f1f69a40b554ba8be22cffbd4df46df in term 1.
I20260620 22:01:16.779032 21135 raft_consensus.cc:2454] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f1f69a40b554ba8be22cffbd4df46df in term 1.
I20260620 22:01:16.779222 20937 leader_election.cc:302] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f1f69a40b554ba8be22cffbd4df46df, c692fd88b81549ce87ec0f34661d3b36; no voters:
I20260620 22:01:16.779350 21336 raft_consensus.cc:2781] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:16.779569 21336 raft_consensus.cc:686] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 1 LEADER]: Becoming Leader. State: Replica: 4f1f69a40b554ba8be22cffbd4df46df, State: Running, Role: LEADER
I20260620 22:01:16.779682 21336 consensus_queue.cc:237] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:16.780532 20856 catalog_manager.cc:5495] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f1f69a40b554ba8be22cffbd4df46df (127.6.241.129). New cstate: current_term: 1 leader_uuid: "4f1f69a40b554ba8be22cffbd4df46df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } health_report { overall_health: UNKNOWN } } }
I20260620 22:01:16.802304 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.799655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.6.241.129:38673)
0620 22:01:16.799719 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:16.799721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:16.799730 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:01:16.799766 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:16.799767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:16.799785 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:16.799851 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:01:16.800035 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.800518 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.801396 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:16.801633 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:16.801696 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:16.801914 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:01:16.802037 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:01:16.802146 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:16.802164 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 22:01:16.802186 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:01:16.804211 21269 raft_consensus.cc:1261] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [term 1 FOLLOWER]: Refusing update from remote peer 4f1f69a40b554ba8be22cffbd4df46df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:16.804252 21135 raft_consensus.cc:1261] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Refusing update from remote peer 4f1f69a40b554ba8be22cffbd4df46df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:16.804653 21336 consensus_queue.cc:1035] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [LEADER]: Connected to new peer: Peer: permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:16.804795 21339 consensus_queue.cc:1035] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:16.807165 21342 mvcc.cc:205] Tried to move back new op lower bound from 7299042823387254784 to 7299042823290064896. Current Snapshot: MvccSnapshot[applied={T|T < 7299042823387254784}]
I20260620 22:01:16.807274 21344 mvcc.cc:205] Tried to move back new op lower bound from 7299042823387254784 to 7299042823290064896. Current Snapshot: MvccSnapshot[applied={T|T < 7299042823387254784}]
I20260620 22:01:16.809345 20856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:16.809435 20856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:16.810288 20856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Sending DeleteTablet for 3 replicas of tablet 990df9ec657f458fa3e066aa0dd34c5d
I20260620 22:01:16.810564 21249 tablet_service.cc:1514] Processing DeleteTablet for tablet 990df9ec657f458fa3e066aa0dd34c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:16 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:49460
I20260620 22:01:16.810660 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 20918
I20260620 22:01:16.810567 21248 tablet_service.cc:1514] Processing DeleteTablet for tablet 990df9ec657f458fa3e066aa0dd34c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:16 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:49460
I20260620 22:01:16.810817 20981 tablet_service.cc:1514] Processing DeleteTablet for tablet 990df9ec657f458fa3e066aa0dd34c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:16 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:45272
I20260620 22:01:16.811384 21351 tablet_replica.cc:330] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: stopping tablet replica
I20260620 22:01:16.811489 21351 raft_consensus.cc:2229] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:16.811591 21351 raft_consensus.cc:2258] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:16.811977 21351 ts_tablet_manager.cc:1905] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:16.813935 21351 ts_tablet_manager.cc:1918] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:16.814015 21351 log.cc:1196] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/990df9ec657f458fa3e066aa0dd34c5d
I20260620 22:01:16.814152 21115 tablet_service.cc:1514] Processing DeleteTablet for tablet 990df9ec657f458fa3e066aa0dd34c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:16 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:60686
I20260620 22:01:16.814301 21351 ts_tablet_manager.cc:1939] T 990df9ec657f458fa3e066aa0dd34c5d P 19345c3de51a40f7ac34406220433d8b: Deleting consensus metadata
I20260620 22:01:16.814419 21352 tablet_replica.cc:330] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: stopping tablet replica
I20260620 22:01:16.814548 21352 raft_consensus.cc:2229] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:16.814697 21352 raft_consensus.cc:2258] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:16.815114 21352 ts_tablet_manager.cc:1905] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:16.817047 21352 ts_tablet_manager.cc:1918] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:16.817137 21352 log.cc:1196] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/990df9ec657f458fa3e066aa0dd34c5d
I20260620 22:01:16.817399 21352 ts_tablet_manager.cc:1939] T 990df9ec657f458fa3e066aa0dd34c5d P c692fd88b81549ce87ec0f34661d3b36: Deleting consensus metadata
I20260620 22:01:16.818406 20843 catalog_manager.cc:4841] TS c692fd88b81549ce87ec0f34661d3b36 (127.6.241.130:35823): tablet 990df9ec657f458fa3e066aa0dd34c5d (table test-table [id=925f02f077e047fa8949f250c17a36c0]) successfully deleted
I20260620 22:01:16.818488 20843 catalog_manager.cc:4841] TS 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437): tablet 990df9ec657f458fa3e066aa0dd34c5d (table test-table [id=925f02f077e047fa8949f250c17a36c0]) successfully deleted
W20260620 22:01:16.818560 20843 catalog_manager.cc:4808] TS 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437): delete failed for tablet 990df9ec657f458fa3e066aa0dd34c5d because the tablet was not found. No further retry: Not found: Tablet not found: 990df9ec657f458fa3e066aa0dd34c5d
W20260620 22:01:16.818786 20840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:38673 (error 108)
I20260620 22:01:16.819012 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 21052
I20260620 22:01:16.819669 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:16.819151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:38673 (local address 127.0.0.1:45298)
0620 22:01:16.819589 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:16.819627 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417}
W20260620 22:01:16.821200 20840 catalog_manager.cc:4584] TS 4f1f69a40b554ba8be22cffbd4df46df (127.6.241.129:38673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 990df9ec657f458fa3e066aa0dd34c5d: Network error: Client connection negotiation failed: client connection to 127.6.241.129:38673: connect: Connection refused (error 111)
I20260620 22:01:16.824409 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 21186
I20260620 22:01:16.828882 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 20824
I20260620 22:01:16.834441 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:32893
--webserver_interface=127.6.241.190
--webserver_port=44175
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:16.916704 21354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:16.916962 21354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:16.917003 21354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:16.918228 21354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:16.918303 21354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:16.918326 21354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:16.918349 21354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:16.918376 21354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:16.919955 21354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:32893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=44175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:16.920248 21354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:16.920490 21354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:16.929303 21363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:16.929328 21360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:16.929567 21354 server_base.cc:1034] running on GCE node
W20260620 22:01:16.929306 21361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:16.930224 21354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:16.930581 21354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:16.931800 21354 hybrid_clock.cc:648] HybridClock initialized: now 1781992876931789 us; error 30 us; skew 500 ppm
Jun 20 22:01:16 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992876, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:16.934266 21354 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:01:16.934820 21354 webserver.cc:460] Webserver started at http://127.6.241.190:44175/ using document root <none> and password file <none>
I20260620 22:01:16.935068 21354 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:16.935163 21354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:16.936595 21354 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:16.937279 21370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:16.937471 21354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:16.937574 21354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f4c6a9a580864beba2d1584e798973de"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "e5e348323b33520dbde9fa4b2f7dc073"
server_key_iv: "f72986563477c02f5375e112214f4eb1"
server_key_version: "encryptionkey@0"
I20260620 22:01:16.937983 21354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:16.967106 21354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:16.969219 21354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:16.969391 21354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:16.973693 21354 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:32893
I20260620 22:01:16.973749 21422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:32893 every 8 connection(s)
I20260620 22:01:16.974179 21354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:01:16.975914 21423 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:01:16.976657 21423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Bootstrap starting.
I20260620 22:01:16.979717 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 21354
I20260620 22:01:16.980060 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:38673
--local_ip_for_outbound_sockets=127.6.241.129
--tserver_master_addrs=127.6.241.190:32893
--webserver_port=38945
--webserver_interface=127.6.241.129
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:16.985589 21423 log.cc:824] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:16.987612 21423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:16.987927 21423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Bootstrap complete.
I20260620 22:01:16.990214 21423 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:16.990433 21423 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c6a9a580864beba2d1584e798973de, State: Initialized, Role: FOLLOWER
I20260620 22:01:16.990635 21423 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:16.990716 21423 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:01:16.990769 21423 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:01:16.990820 21423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:01:16.991852 21423 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:16.991940 21423 leader_election.cc:302] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c6a9a580864beba2d1584e798973de; no voters:
I20260620 22:01:16.992162 21423 leader_election.cc:288] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:01:16.992233 21427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:01:16.992372 21427 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 2 LEADER]: Becoming Leader. State: Replica: f4c6a9a580864beba2d1584e798973de, State: Running, Role: LEADER
I20260620 22:01:16.992447 21423 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:16.992475 21427 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:16.992745 21427 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c6a9a580864beba2d1584e798973de. Latest consensus state: current_term: 2 leader_uuid: "f4c6a9a580864beba2d1584e798973de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } } }
I20260620 22:01:16.992811 21427 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:16.992937 21428 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f4c6a9a580864beba2d1584e798973de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } } }
I20260620 22:01:16.992993 21428 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:16.993079 21432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:01:16.994046 21432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:01:16.994144 21432 catalog_manager.cc:1259] Loaded cluster ID: a1a5f58dc47b4141bc25336e5ffc217a
I20260620 22:01:16.994177 21432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:01:16.994428 21432 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 22:01:16.999522 21432 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 22:01:17.002170 21432 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:01:17.002379 21432 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Loaded TSK: 0
I20260620 22:01:17.002622 21432 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:01:17.066545 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:17.066788 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:17.066813 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:17.068064 21425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:17.068116 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:17.068145 21425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:01:17.069644 21425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:38673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=38945
--tserver_master_addrs=127.6.241.190:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:17.069903 21425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:17.070165 21425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:17.076301 21451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:17.076480 21425 server_base.cc:1034] running on GCE node
W20260620 22:01:17.076304 21450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:17.076315 21453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:17.076814 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:17.077101 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:17.078262 21425 hybrid_clock.cc:648] HybridClock initialized: now 1781992877078251 us; error 27 us; skew 500 ppm
Jun 20 22:01:17 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:17.081063 21425 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:01:17.081632 21425 webserver.cc:460] Webserver started at http://127.6.241.129:38945/ using document root <none> and password file <none>
I20260620 22:01:17.081897 21425 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:17.081981 21425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:17.083755 21425 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:17.084504 21460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.084715 21425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:17.084818 21425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "4f1f69a40b554ba8be22cffbd4df46df"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "4915a7973f60b5ee604cd7004cb54a44"
server_key_iv: "9d5be450a1a58f7b463778f1b56aae2b"
server_key_version: "encryptionkey@0"
I20260620 22:01:17.085287 21425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:17.118422 21425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:17.120934 21425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:17.121097 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:17.121387 21425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:17.121946 21467 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:01:17.123554 21425 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:01:17.123605 21425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:01:17.123641 21425 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:01:17.124320 21425 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:01:17.124364 21425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:01:17.124408 21467 tablet_bootstrap.cc:492] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Bootstrap starting.
I20260620 22:01:17.132171 21425 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:38673
I20260620 22:01:17.132282 21574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:38673 every 8 connection(s)
I20260620 22:01:17.132731 21425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:01:17.135847 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 21425
I20260620 22:01:17.136101 21467 log.cc:824] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:17.136140 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:35823
--local_ip_for_outbound_sockets=127.6.241.130
--tserver_master_addrs=127.6.241.190:32893
--webserver_port=34293
--webserver_interface=127.6.241.130
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:17.138513 21467 tablet_bootstrap.cc:492] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:17.138845 21467 tablet_bootstrap.cc:492] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Bootstrap complete.
I20260620 22:01:17.139993 21467 ts_tablet_manager.cc:1397] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
Jun 20 22:01:17 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:17.141700 21467 raft_consensus.cc:348] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:17.142122 21467 raft_consensus.cc:729] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1f69a40b554ba8be22cffbd4df46df, State: Initialized, Role: FOLLOWER
I20260620 22:01:17.142273 21467 consensus_queue.cc:260] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } }
I20260620 22:01:17.142585 21467 ts_tablet_manager.cc:1428] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260620 22:01:17.145399 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.133393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:60773)
0620 22:01:17.133582 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.133650 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:17.134407 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:17.134964 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:17.134970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:17.135472 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:17.135756 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.135763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.137451 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.137452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.138725 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.138729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.139363 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.139364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.139486 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.140072 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:17.140083 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:17.141617 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:17.143764 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:17.143771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:17.143773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:17.143993 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:17.144281 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:17.144282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:17.144282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:17.144382 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:17.144789 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:17.144795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:17.144922 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 22:01:17.145028 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 22:01:17.145206 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260620 22:01:17.145401 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.133553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:60773 (local address 127.6.241.190:32893)
0620 22:01:17.133712 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.133721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.133759 (+ 38us) server_negotiation.cc:407] Connection header received
0620 22:01:17.134715 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.134720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.134763 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.134872 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:17.135891 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.137320 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.138809 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.139250 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.139383 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.141727 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:17.141730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:17.141731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:17.141743 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:17.143658 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.144093 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.144096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.144097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.144205 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.144470 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.144471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.144471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.144704 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:17.144793 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.145078 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.145206 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260620 22:01:17.146385 21575 heartbeater.cc:347] Connected to a master server at 127.6.241.190:32893
I20260620 22:01:17.146489 21575 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:17.146742 21575 heartbeater.cc:511] Master 127.6.241.190:32893 requested a full tablet report, sending...
I20260620 22:01:17.147508 21387 ts_manager.cc:194] Registered new tserver with Master: 4f1f69a40b554ba8be22cffbd4df46df (127.6.241.129:38673)
I20260620 22:01:17.148705 21387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:60773
I20260620 22:01:17.149508 21575 heartbeater.cc:503] Master 127.6.241.190:32893 was elected leader, sending a full tablet report...
Jun 20 22:01:17 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992876, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 22:01:17.156338 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.148189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.6.241.129:38673)
0620 22:01:17.148527 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.148530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.148538 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:01:17.148561 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.148563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.148573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.148651 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:17.149162 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.150246 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.151084 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.151529 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.151600 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.153624 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:17.153630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:17.153630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:17.153636 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:17.155177 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.155608 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.155610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.155610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.155660 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.155844 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.155845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.155845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.155968 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:17.156026 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.156203 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.156276 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":68,"threads_started":1}
I20260620 22:01:17.156383 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.148145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:38673 (local address 127.0.0.1:45302)
0620 22:01:17.148360 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.148385 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:17.148435 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:17.148653 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:17.148658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:17.149025 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:17.149078 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.149081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.150322 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.150323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.151013 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.151015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.151611 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.151612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.151661 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.152132 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:17.152137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:17.153534 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:17.155251 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:17.155257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:17.155259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:17.155518 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:17.155722 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:17.155723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:17.155723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:17.155788 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:17.156029 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:17.156034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:17.156086 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 22:01:17.156169 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:01:17.156281 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260620 22:01:17.156836 21509 tablet_service.cc:1514] Processing DeleteTablet for tablet 990df9ec657f458fa3e066aa0dd34c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:16 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:45302
I20260620 22:01:17.157333 21584 tablet_replica.cc:330] stopping tablet replica
I20260620 22:01:17.157449 21584 raft_consensus.cc:2229] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:17.157516 21584 raft_consensus.cc:2258] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:17.157800 21584 ts_tablet_manager.cc:1905] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:17.159904 21584 ts_tablet_manager.cc:1918] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:17.159965 21584 log.cc:1196] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/990df9ec657f458fa3e066aa0dd34c5d
I20260620 22:01:17.160212 21584 ts_tablet_manager.cc:1939] T 990df9ec657f458fa3e066aa0dd34c5d P 4f1f69a40b554ba8be22cffbd4df46df: Deleting consensus metadata
I20260620 22:01:17.160707 21371 catalog_manager.cc:4841] TS 4f1f69a40b554ba8be22cffbd4df46df (127.6.241.129:38673): tablet 990df9ec657f458fa3e066aa0dd34c5d (table test-table [id=925f02f077e047fa8949f250c17a36c0]) successfully deleted
W20260620 22:01:17.223059 21580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:17.223318 21580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:17.223358 21580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:17.224551 21580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:17.224627 21580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:17.224673 21580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:01:17.226174 21580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:35823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=34293
--tserver_master_addrs=127.6.241.190:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:17.226444 21580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:17.226739 21580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:17.231755 21589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:17.231813 21592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:17.231753 21590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:17.232010 21580 server_base.cc:1034] running on GCE node
I20260620 22:01:17.232257 21580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:17.232549 21580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:17.233714 21580 hybrid_clock.cc:648] HybridClock initialized: now 1781992877233711 us; error 39 us; skew 500 ppm
Jun 20 22:01:17 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:17.236373 21580 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:01:17.236840 21580 webserver.cc:460] Webserver started at http://127.6.241.130:34293/ using document root <none> and password file <none>
I20260620 22:01:17.237066 21580 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:17.237151 21580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:17.238518 21580 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:17.239177 21599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.239346 21580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.239442 21580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "c692fd88b81549ce87ec0f34661d3b36"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "112c3dce80301f9856c569e5801636a4"
server_key_iv: "8dd8367b29bad024248e54132022c12c"
server_key_version: "encryptionkey@0"
I20260620 22:01:17.239825 21580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:17.276544 21580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:17.278416 21580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:17.278618 21580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:17.278872 21580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:17.279198 21580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:17.279253 21580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.279296 21580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:17.279340 21580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.285562 21580 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:35823
I20260620 22:01:17.285657 21712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:35823 every 8 connection(s)
I20260620 22:01:17.285991 21580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:01:17.293006 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 21580
I20260620 22:01:17.293411 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:44437
--local_ip_for_outbound_sockets=127.6.241.131
--tserver_master_addrs=127.6.241.190:32893
--webserver_port=35901
--webserver_interface=127.6.241.131
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:01:17 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:17.299644 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.286792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:50129 (local address 127.6.241.190:32893)
0620 22:01:17.286874 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.286877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.287082 (+ 205us) server_negotiation.cc:407] Connection header received
0620 22:01:17.287554 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.287555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.287565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.287628 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:17.288525 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.289039 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.290083 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.290277 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.290330 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.296082 (+ 5752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:17.296083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:17.296084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:17.296093 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:17.297880 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.298297 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.298299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.298299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.298366 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.298747 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.298748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.298748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.298874 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:17.298935 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.299301 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.299419 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:01:17.299999 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.286790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:50129)
0620 22:01:17.286961 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.286998 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:17.287471 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:17.287683 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:17.287686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:17.288252 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:17.288446 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.288451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.289120 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.289121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.289992 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.289995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.290346 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.290347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.290447 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.294349 (+ 3902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:17.294361 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:17.295971 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:17.297980 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:17.297986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:17.297988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:17.298217 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:17.298458 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:17.298459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:17.298460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:17.298664 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:17.298949 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:17.298956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:17.299150 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 22:01:17.299285 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 22:01:17.299699 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260620 22:01:17.300591 21713 heartbeater.cc:347] Connected to a master server at 127.6.241.190:32893
I20260620 22:01:17.300705 21713 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:17.300895 21713 heartbeater.cc:511] Master 127.6.241.190:32893 requested a full tablet report, sending...
I20260620 22:01:17.301318 21387 ts_manager.cc:194] Registered new tserver with Master: c692fd88b81549ce87ec0f34661d3b36 (127.6.241.130:35823)
I20260620 22:01:17.301697 21387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:50129
W20260620 22:01:17.376718 21716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:17.376938 21716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:17.376962 21716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:17.378166 21716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:17.378209 21716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:17.378269 21716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:01:17.379729 21716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:44437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=35901
--tserver_master_addrs=127.6.241.190:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:17.379988 21716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:17.380234 21716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:17.385284 21724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:17.385284 21721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:17.385344 21722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:17.385304 21716 server_base.cc:1034] running on GCE node
I20260620 22:01:17.385726 21716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:17.385998 21716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:17.387154 21716 hybrid_clock.cc:648] HybridClock initialized: now 1781992877387153 us; error 37 us; skew 500 ppm
Jun 20 22:01:17 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:17.389600 21716 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:01:17.390025 21716 webserver.cc:460] Webserver started at http://127.6.241.131:35901/ using document root <none> and password file <none>
I20260620 22:01:17.390323 21716 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:17.390377 21716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:17.391806 21716 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:17.392467 21731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.392683 21716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:17.392776 21716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "19345c3de51a40f7ac34406220433d8b"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "540cf96fb6216095f88cfdf49e33b15c"
server_key_iv: "fe7efaa812e25dcf464df2329ce2ffbd"
server_key_version: "encryptionkey@0"
I20260620 22:01:17.393190 21716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:17.407687 21716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:17.409885 21716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:17.410000 21716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:17.410192 21716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:17.410439 21716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:17.410468 21716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.410490 21716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:17.410503 21716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.416826 21716 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:44437
I20260620 22:01:17.416904 21844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:44437 every 8 connection(s)
I20260620 22:01:17.417228 21716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:01:17.418768 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 21716
Jun 20 22:01:17 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:17.426378 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.419115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.6.241.190:32893)
0620 22:01:17.419428 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.419431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.419439 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:01:17.419459 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.419460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.419471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.419538 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:17.420144 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.420642 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.421356 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.421566 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.421648 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.423521 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:17.423522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:17.423522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:17.423529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:17.425240 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.425593 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.425594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.425594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.425653 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.425853 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.425854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.425855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.425970 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:17.426148 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.426187 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.426267 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":52,"threads_started":1}
I20260620 22:01:17.426692 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.417963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:57297 (local address 127.6.241.190:32893)
0620 22:01:17.418030 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.418032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.418252 (+ 220us) server_negotiation.cc:407] Connection header received
0620 22:01:17.418864 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.418866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.418877 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.418922 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:17.420087 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.420599 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.421574 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.421772 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.421844 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.423886 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:17.423887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:17.423887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:17.423895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:17.425299 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.425701 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.425702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.425702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.425762 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:17.426008 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:17.426009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:17.426010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:17.426125 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:17.426202 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.426499 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.426576 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:01:17.426910 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.417948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:57297)
0620 22:01:17.418128 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.418163 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:17.418756 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:17.418992 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:17.418996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:17.419785 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:17.419979 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.419983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.420688 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.420689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.421476 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.421478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.421863 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.421864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.421991 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.422381 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:17.422389 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:17.423789 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:17.425380 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:17.425386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:17.425389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:17.425612 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:17.425827 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:17.425828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:17.425829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:17.425941 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:17.426196 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:17.426203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:17.426369 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 22:01:17.426479 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 22:01:17.426674 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260620 22:01:17.427418 21845 heartbeater.cc:347] Connected to a master server at 127.6.241.190:32893
I20260620 22:01:17.427546 21845 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:17.427713 21845 heartbeater.cc:511] Master 127.6.241.190:32893 requested a full tablet report, sending...
I20260620 22:01:17.428042 21386 ts_manager.cc:194] Registered new tserver with Master: 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437)
I20260620 22:01:17.428416 21386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:57297
I20260620 22:01:17.438949 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:01:17.441823 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.439302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.6.241.190:32893)
0620 22:01:17.439352 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.439354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.439444 (+ 90us) server_negotiation.cc:407] Connection header received
0620 22:01:17.439516 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.439516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.439552 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.439617 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:01:17.439743 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.440270 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.441075 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.441269 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.441337 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.441516 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:01:17.441622 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:01:17.441685 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.441735 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.441757 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:01:17.442108 21386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:01:17.444005 21386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:01:17.446146 21509 tablet_service.cc:1467] Processing CreateTablet for tablet bed22d09c9b14fc69e2f4e0040f7d150 (DEFAULT_TABLE table=test-table [id=23180149e9874a499d8e24bd433019ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:17.446372 21509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed22d09c9b14fc69e2f4e0040f7d150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:17.447758 21467 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Bootstrap starting.
I20260620 22:01:17.448113 21467 tablet_bootstrap.cc:654] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:17.448668 21467 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: No bootstrap required, opened a new log
I20260620 22:01:17.448729 21467 ts_tablet_manager.cc:1397] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:01:17.449087 21467 raft_consensus.cc:348] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:17.449167 21467 raft_consensus.cc:374] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:17.449220 21467 raft_consensus.cc:729] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1f69a40b554ba8be22cffbd4df46df, State: Initialized, Role: FOLLOWER
I20260620 22:01:17.449266 21467 consensus_queue.cc:260] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:17.449362 21467 ts_tablet_manager.cc:1428] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:01:17.450253 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.445876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.6.241.130:35823)
0620 22:01:17.446099 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.446104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.446114 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:01:17.446160 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.446162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.446179 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.446263 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:17.446409 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.447558 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.449004 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.449890 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.449950 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.450011 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.450125 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.450154 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
I20260620 22:01:17.450238 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.445777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35823 (local address 127.0.0.1:60698)
0620 22:01:17.445913 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.445925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:17.445964 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:17.446287 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:17.446288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:17.446290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:17.446335 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.446336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.447678 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.447679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.448912 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.448914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.449960 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.449961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.450036 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.450055 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 22:01:17.450115 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:01:17.450168 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 22:01:17.450322 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.445890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:44437 (local address 127.0.0.1:49468)
0620 22:01:17.446206 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.446216 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:17.446458 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:17.446764 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:17.446765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:17.446767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:17.446806 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.446808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.447985 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.447986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.449026 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.449028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.450067 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.450068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.450137 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.450139 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:01:17.450188 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:01:17.450221 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":57,"threads_started":1}
I20260620 22:01:17.450356 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.446015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.6.241.131:44437)
0620 22:01:17.446206 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.446211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.446389 (+ 178us) server_negotiation.cc:407] Connection header received
0620 22:01:17.446561 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.446564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.446581 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.446683 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:17.446878 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.447879 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.449137 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.449906 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.449955 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.450008 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.450217 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.450262 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260620 22:01:17.450989 21647 tablet_service.cc:1467] Processing CreateTablet for tablet bed22d09c9b14fc69e2f4e0040f7d150 (DEFAULT_TABLE table=test-table [id=23180149e9874a499d8e24bd433019ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:17.451327 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed22d09c9b14fc69e2f4e0040f7d150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:17.453899 21855 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Bootstrap starting.
I20260620 22:01:17.454344 21779 tablet_service.cc:1467] Processing CreateTablet for tablet bed22d09c9b14fc69e2f4e0040f7d150 (DEFAULT_TABLE table=test-table [id=23180149e9874a499d8e24bd433019ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:17.454692 21855 tablet_bootstrap.cc:654] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:17.454723 21779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed22d09c9b14fc69e2f4e0040f7d150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:17.455188 21855 log.cc:824] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:17.455886 21855 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: No bootstrap required, opened a new log
I20260620 22:01:17.455940 21855 ts_tablet_manager.cc:1397] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:01:17.457265 21855 raft_consensus.cc:348] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:17.457517 21857 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Bootstrap starting.
I20260620 22:01:17.457571 21855 raft_consensus.cc:374] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:17.457612 21855 raft_consensus.cc:729] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c692fd88b81549ce87ec0f34661d3b36, State: Initialized, Role: FOLLOWER
I20260620 22:01:17.457737 21855 consensus_queue.cc:260] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:17.458058 21855 ts_tablet_manager.cc:1428] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:01:17.458088 21713 heartbeater.cc:503] Master 127.6.241.190:32893 was elected leader, sending a full tablet report...
I20260620 22:01:17.458215 21857 tablet_bootstrap.cc:654] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:17.458751 21857 log.cc:824] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:17.459653 21857 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: No bootstrap required, opened a new log
I20260620 22:01:17.459707 21857 ts_tablet_manager.cc:1397] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:01:17.460983 21857 raft_consensus.cc:348] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:17.461109 21857 raft_consensus.cc:374] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:17.461164 21857 raft_consensus.cc:729] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19345c3de51a40f7ac34406220433d8b, State: Initialized, Role: FOLLOWER
I20260620 22:01:17.461290 21857 consensus_queue.cc:260] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:17.461570 21845 heartbeater.cc:503] Master 127.6.241.190:32893 was elected leader, sending a full tablet report...
I20260620 22:01:17.461570 21857 ts_tablet_manager.cc:1428] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260620 22:01:17.536553 21714 tablet.cc:2369] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:01:17.633472 21576 tablet.cc:2369] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:01:17.667807 21846 tablet.cc:2369] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:01:17.711356 21860 raft_consensus.cc:482] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:01:17.711493 21860 raft_consensus.cc:504] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:17.711948 21860 leader_election.cc:288] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c692fd88b81549ce87ec0f34661d3b36 (127.6.241.130:35823), 4f1f69a40b554ba8be22cffbd4df46df (127.6.241.129:38673)
I20260620 22:01:17.714852 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.712156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35823 (local address 127.6.241.131:34197)
0620 22:01:17.712220 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.712231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:17.712276 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:17.712450 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:17.712451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:17.712452 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:17.712534 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.712536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.713194 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.713195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.713955 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.713957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.714534 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.714535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.714673 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.714677 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:01:17.714734 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:01:17.714777 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 22:01:17.714852 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.712222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:34197 (local address 127.6.241.130:35823)
0620 22:01:17.712275 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.712277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.712289 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:01:17.712359 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.712361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.712373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.712446 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:17.712614 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.713111 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.714056 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.714440 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.714535 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.714544 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.714750 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.714792 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:01:17.715018 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.712265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:48031 (local address 127.6.241.129:38673)
0620 22:01:17.712399 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.712401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.712411 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:01:17.712517 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.712518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.712555 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.712648 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:17.712811 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.713282 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.714206 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.714649 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.714741 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.714828 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.714872 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.714915 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260620 22:01:17.715180 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.712194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:38673 (local address 127.6.241.131:48031)
0620 22:01:17.712356 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.712368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:17.712428 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:17.712651 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:17.712652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:17.712654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:17.712730 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.712732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.713368 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.713369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.714115 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:17.714117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.714737 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.714738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:17.714803 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.714808 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:01:17.714864 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:01:17.715037 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260620 22:01:17.715205 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed22d09c9b14fc69e2f4e0040f7d150" candidate_uuid: "19345c3de51a40f7ac34406220433d8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c692fd88b81549ce87ec0f34661d3b36" is_pre_election: true
I20260620 22:01:17.715363 21667 raft_consensus.cc:2454] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19345c3de51a40f7ac34406220433d8b in term 0.
I20260620 22:01:17.715482 21529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed22d09c9b14fc69e2f4e0040f7d150" candidate_uuid: "19345c3de51a40f7ac34406220433d8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f1f69a40b554ba8be22cffbd4df46df" is_pre_election: true
I20260620 22:01:17.715642 21529 raft_consensus.cc:2454] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19345c3de51a40f7ac34406220433d8b in term 0.
I20260620 22:01:17.715627 21735 leader_election.cc:302] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19345c3de51a40f7ac34406220433d8b, c692fd88b81549ce87ec0f34661d3b36; no voters:
I20260620 22:01:17.715775 21860 raft_consensus.cc:2781] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:01:17.715842 21860 raft_consensus.cc:482] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:01:17.715878 21860 raft_consensus.cc:3037] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:17.716624 21860 raft_consensus.cc:504] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:17.716773 21860 leader_election.cc:288] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [CANDIDATE]: Term 1 election: Requested vote from peers c692fd88b81549ce87ec0f34661d3b36 (127.6.241.130:35823), 4f1f69a40b554ba8be22cffbd4df46df (127.6.241.129:38673)
I20260620 22:01:17.716907 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed22d09c9b14fc69e2f4e0040f7d150" candidate_uuid: "19345c3de51a40f7ac34406220433d8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c692fd88b81549ce87ec0f34661d3b36"
I20260620 22:01:17.716923 21529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed22d09c9b14fc69e2f4e0040f7d150" candidate_uuid: "19345c3de51a40f7ac34406220433d8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f1f69a40b554ba8be22cffbd4df46df"
I20260620 22:01:17.716992 21529 raft_consensus.cc:3037] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:17.716993 21667 raft_consensus.cc:3037] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:17.717825 21667 raft_consensus.cc:2454] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19345c3de51a40f7ac34406220433d8b in term 1.
I20260620 22:01:17.717825 21529 raft_consensus.cc:2454] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19345c3de51a40f7ac34406220433d8b in term 1.
I20260620 22:01:17.718014 21732 leader_election.cc:302] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19345c3de51a40f7ac34406220433d8b, 4f1f69a40b554ba8be22cffbd4df46df; no voters:
I20260620 22:01:17.718127 21860 raft_consensus.cc:2781] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:17.718298 21860 raft_consensus.cc:686] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 1 LEADER]: Becoming Leader. State: Replica: 19345c3de51a40f7ac34406220433d8b, State: Running, Role: LEADER
I20260620 22:01:17.718396 21860 consensus_queue.cc:237] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:17.719210 21386 catalog_manager.cc:5495] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 19345c3de51a40f7ac34406220433d8b (127.6.241.131). New cstate: current_term: 1 leader_uuid: "19345c3de51a40f7ac34406220433d8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } health_report { overall_health: UNKNOWN } } }
I20260620 22:01:17.733568 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.730377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.6.241.131:44437)
0620 22:01:17.730440 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:17.730442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:17.730452 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:01:17.730511 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:17.730512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:17.730531 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:17.730635 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:01:17.730776 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.731548 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.732606 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:17.732888 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:17.732942 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:17.733125 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:01:17.733246 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:01:17.733312 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:17.733452 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 22:01:17.733471 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:01:17.738704 21667 raft_consensus.cc:1261] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Refusing update from remote peer 19345c3de51a40f7ac34406220433d8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:17.738731 21529 raft_consensus.cc:1261] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Refusing update from remote peer 19345c3de51a40f7ac34406220433d8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:17.739061 21863 consensus_queue.cc:1035] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:17.739181 21860 consensus_queue.cc:1035] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:17.741402 21866 mvcc.cc:205] Tried to move back new op lower bound from 7299042827214786560 to 7299042827135098880. Current Snapshot: MvccSnapshot[applied={T|T < 7299042827214786560}]
I20260620 22:01:17.741460 21867 mvcc.cc:205] Tried to move back new op lower bound from 7299042827214786560 to 7299042827135098880. Current Snapshot: MvccSnapshot[applied={T|T < 7299042827214786560}]
I20260620 22:01:17.741997 21868 mvcc.cc:205] Tried to move back new op lower bound from 7299042827214786560 to 7299042827135098880. Current Snapshot: MvccSnapshot[applied={T|T < 7299042827214786560}]
I20260620 22:01:17.743835 21387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:17.743927 21387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:17.744670 21387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Sending DeleteTablet for 3 replicas of tablet bed22d09c9b14fc69e2f4e0040f7d150
I20260620 22:01:17.745028 21779 tablet_service.cc:1514] Processing DeleteTablet for tablet bed22d09c9b14fc69e2f4e0040f7d150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:17 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:49468
I20260620 22:01:17.745087 21647 tablet_service.cc:1514] Processing DeleteTablet for tablet bed22d09c9b14fc69e2f4e0040f7d150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:17 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:60698
I20260620 22:01:17.745302 21876 tablet_replica.cc:330] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: stopping tablet replica
I20260620 22:01:17.745301 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 21425
I20260620 22:01:17.746702 21875 tablet_replica.cc:330] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: stopping tablet replica
I20260620 22:01:17.746798 21875 raft_consensus.cc:2229] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:01:17.746910 21875 raft_consensus.cc:2258] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:17.747298 21875 ts_tablet_manager.cc:1905] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:17.749104 21875 ts_tablet_manager.cc:1918] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:17.749166 21875 log.cc:1196] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/bed22d09c9b14fc69e2f4e0040f7d150
I20260620 22:01:17.749445 21875 ts_tablet_manager.cc:1939] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Deleting consensus metadata
I20260620 22:01:17.749831 21374 catalog_manager.cc:4841] TS 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437): tablet bed22d09c9b14fc69e2f4e0040f7d150 (table test-table [id=23180149e9874a499d8e24bd433019ff]) successfully deleted
W20260620 22:01:17.750159 21371 connection.cc:535] client connection to 127.6.241.129:38673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 22:01:17.750248 21371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:01:17.750780 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.750558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:38673 (local address 127.0.0.1:45306)
0620 22:01:17.750660 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.750738 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 22:01:17.751152 21876 raft_consensus.cc:2229] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:17.751283 21876 raft_consensus.cc:2258] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 22:01:17.750911 21371 catalog_manager.cc:4584] TS 4f1f69a40b554ba8be22cffbd4df46df (127.6.241.129:38673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bed22d09c9b14fc69e2f4e0040f7d150: Network error: Client connection negotiation failed: client connection to 127.6.241.129:38673: connect: Connection refused (error 111)
I20260620 22:01:17.751447 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 21580
I20260620 22:01:17.751685 21876 ts_tablet_manager.cc:1905] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:17.757465 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 21716
I20260620 22:01:17.757548 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:17.757412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35823 (local address 127.0.0.1:60704)
0620 22:01:17.757476 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:17.757508 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.130:35823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 22:01:17.761415 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 21354
I20260620 22:01:17.766428 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:32893
--webserver_interface=127.6.241.190
--webserver_port=44175
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:17.846426 21878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:17.846654 21878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:17.846679 21878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:17.847764 21878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:17.847797 21878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:17.847812 21878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:17.847824 21878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:17.847836 21878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:17.849185 21878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:32893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=44175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:17.849375 21878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:17.849537 21878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:01:17.854332 21878 server_base.cc:1034] running on GCE node
W20260620 22:01:17.854271 21886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:17.854266 21884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:17.854290 21883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:17.854795 21878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:17.855110 21878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:17.856256 21878 hybrid_clock.cc:648] HybridClock initialized: now 1781992877856234 us; error 34 us; skew 500 ppm
Jun 20 22:01:17 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:17.858866 21878 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:01:17.859323 21878 webserver.cc:460] Webserver started at http://127.6.241.190:44175/ using document root <none> and password file <none>
I20260620 22:01:17.859500 21878 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:17.859541 21878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:17.860781 21878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:17.861394 21893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.861577 21878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:17.861649 21878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f4c6a9a580864beba2d1584e798973de"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "e5e348323b33520dbde9fa4b2f7dc073"
server_key_iv: "f72986563477c02f5375e112214f4eb1"
server_key_version: "encryptionkey@0"
I20260620 22:01:17.861975 21878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:17.876611 21878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:17.878957 21878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:17.879069 21878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:17.883101 21878 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:32893
I20260620 22:01:17.883137 21945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:32893 every 8 connection(s)
I20260620 22:01:17.883515 21878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:01:17.885094 21946 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:01:17.885708 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Bootstrap starting.
I20260620 22:01:17.891021 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 21878
I20260620 22:01:17.891355 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:38673
--local_ip_for_outbound_sockets=127.6.241.129
--tserver_master_addrs=127.6.241.190:32893
--webserver_port=38945
--webserver_interface=127.6.241.129
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:17.894874 21946 log.cc:824] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:17.897161 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:17.897497 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de: Bootstrap complete.
I20260620 22:01:17.899823 21946 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:17.900146 21946 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c6a9a580864beba2d1584e798973de, State: Initialized, Role: FOLLOWER
I20260620 22:01:17.900318 21946 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:17.900408 21946 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:01:17.900465 21946 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:01:17.900514 21946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 2 FOLLOWER]: Advancing to term 3
I20260620 22:01:17.901500 21946 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:17.901620 21946 leader_election.cc:302] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c6a9a580864beba2d1584e798973de; no voters:
I20260620 22:01:17.901823 21946 leader_election.cc:288] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 22:01:17.901871 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 3 FOLLOWER]: Leader election won for term 3
I20260620 22:01:17.901991 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [term 3 LEADER]: Becoming Leader. State: Replica: f4c6a9a580864beba2d1584e798973de, State: Running, Role: LEADER
I20260620 22:01:17.902107 21946 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:17.902091 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } }
I20260620 22:01:17.902380 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c6a9a580864beba2d1584e798973de. Latest consensus state: current_term: 3 leader_uuid: "f4c6a9a580864beba2d1584e798973de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } } }
I20260620 22:01:17.902465 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:17.902663 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f4c6a9a580864beba2d1584e798973de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6a9a580864beba2d1584e798973de" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 32893 } } }
I20260620 22:01:17.902737 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c6a9a580864beba2d1584e798973de [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:17.902864 21955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:01:17.903738 21955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:01:17.903855 21955 catalog_manager.cc:1259] Loaded cluster ID: a1a5f58dc47b4141bc25336e5ffc217a
I20260620 22:01:17.903909 21955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 22:01:17.904313 21955 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f110018b9ad google::LogMessage::Fail() at ??:0
@ 0x7f110018e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f110018b4de google::LogMessage::Flush() at ??:0
@ 0x7f110018ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f11080b00f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f11080b98b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f11020590d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f110204e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1102eb26db start_thread at ??:0
@ 0x7f10fec8961f clone at ??:0
W20260620 22:01:17.984014 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:17.984247 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:17.984285 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:17.985404 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:17.985466 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:17.985512 21948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:01:17.987089 21948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:38673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=38945
--tserver_master_addrs=127.6.241.190:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:17.987346 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:17.987576 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:17.993623 21982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:17.993657 21983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:17.993743 21948 server_base.cc:1034] running on GCE node
W20260620 22:01:17.993649 21985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:17.994168 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:17.994453 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:17.995625 21948 hybrid_clock.cc:648] HybridClock initialized: now 1781992877995611 us; error 33 us; skew 500 ppm
Jun 20 22:01:17 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:17.998337 21948 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:01:17.998878 21948 webserver.cc:460] Webserver started at http://127.6.241.129:38945/ using document root <none> and password file <none>
I20260620 22:01:17.999136 21948 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:17.999181 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:18.000495 21948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:18.001209 21992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:18.001422 21948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:18.001518 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "4f1f69a40b554ba8be22cffbd4df46df"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "4915a7973f60b5ee604cd7004cb54a44"
server_key_iv: "9d5be450a1a58f7b463778f1b56aae2b"
server_key_version: "encryptionkey@0"
I20260620 22:01:18.001881 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:18.015064 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:18.017341 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:18.017499 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:18.017735 21948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:18.018158 21999 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:01:18.020160 21948 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:01:18.020226 21948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:01:18.020267 21948 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:01:18.020881 21948 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:01:18.020937 21948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:01:18.021004 21999 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Bootstrap starting.
I20260620 22:01:18.027887 21948 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:38673
I20260620 22:01:18.028038 22106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:38673 every 8 connection(s)
I20260620 22:01:18.028327 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:01:18.032431 21999 log.cc:824] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:18.035143 21999 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:18.035447 21999 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Bootstrap complete.
I20260620 22:01:18.036728 21999 ts_tablet_manager.cc:1397] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260620 22:01:18.036827 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 21948
I20260620 22:01:18.037117 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:35823
--local_ip_for_outbound_sockets=127.6.241.130
--tserver_master_addrs=127.6.241.190:32893
--webserver_port=34293
--webserver_interface=127.6.241.130
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:18.038125 21999 raft_consensus.cc:348] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:18.038470 21999 raft_consensus.cc:729] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1f69a40b554ba8be22cffbd4df46df, State: Initialized, Role: FOLLOWER
I20260620 22:01:18.038671 21999 consensus_queue.cc:260] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:18.038949 21999 ts_tablet_manager.cc:1428] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260620 22:01:18.126133 22109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:01:18.029035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:48215)
0620 22:01:18.029231 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:18.029266 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:18.029728 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:18.125861 (+ 96133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
W20260620 22:01:18.126950 22107 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:32893 (0 consecutive failures): Network error: Failed to ping master at 127.6.241.190:32893: Client connection negotiation failed: client connection to 127.6.241.190:32893: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:01:18.127348 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:18.127196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:37049)
0620 22:01:18.127272 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:18.127315 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 22:01:18.127913 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:18.127801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:39977)
0620 22:01:18.127848 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:18.127882 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 22:01:18.128055 22107 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:01:18.135893 22111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:18.136080 22111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:18.136102 22111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:18.137279 22111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:18.137346 22111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:18.137390 22111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:01:18.138957 22111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:35823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=34293
--tserver_master_addrs=127.6.241.190:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:18.139190 22111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:18.139411 22111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:18.144481 22117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:18.144490 22120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:18.144577 22118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:18.144729 22111 server_base.cc:1034] running on GCE node
I20260620 22:01:18.144992 22111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:18.145247 22111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:18.146409 22111 hybrid_clock.cc:648] HybridClock initialized: now 1781992878146408 us; error 41 us; skew 500 ppm
Jun 20 22:01:18 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992878, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:18.148907 22111 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:01:18.149309 22111 webserver.cc:460] Webserver started at http://127.6.241.130:34293/ using document root <none> and password file <none>
I20260620 22:01:18.149533 22111 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:18.149601 22111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:18.150900 22111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:01:18.151607 22127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:18.151774 22111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:01:18.151870 22111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "c692fd88b81549ce87ec0f34661d3b36"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "112c3dce80301f9856c569e5801636a4"
server_key_iv: "8dd8367b29bad024248e54132022c12c"
server_key_version: "encryptionkey@0"
I20260620 22:01:18.152196 22111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:18.177737 22111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:18.179836 22111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:18.180016 22111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:18.180276 22111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:18.180768 22134 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:01:18.182247 22111 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:01:18.182307 22111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:01:18.182351 22111 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:01:18.182986 22111 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:01:18.183043 22111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:01:18.183110 22134 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Bootstrap starting.
I20260620 22:01:18.184660 22134 log.cc:824] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:18.187186 22134 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:18.187474 22134 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Bootstrap complete.
I20260620 22:01:18.187633 22134 ts_tablet_manager.cc:1397] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 22:01:18.188942 22134 raft_consensus.cc:348] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:18.189263 22134 raft_consensus.cc:729] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c692fd88b81549ce87ec0f34661d3b36, State: Initialized, Role: FOLLOWER
I20260620 22:01:18.189481 22134 consensus_queue.cc:260] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:18.189760 22134 ts_tablet_manager.cc:1428] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:01:18.190297 22111 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:35823
I20260620 22:01:18.190335 22243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:35823 every 8 connection(s)
I20260620 22:01:18.190708 22111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:01:18.191658 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:18.191339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:39551)
0620 22:01:18.191482 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:18.191545 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
W20260620 22:01:18.191864 22244 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:32893 (0 consecutive failures): Network error: Failed to ping master at 127.6.241.190:32893: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
I20260620 22:01:18.192147 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:18.192048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:34401)
0620 22:01:18.192108 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:18.192132 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 22:01:18.192370 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:18.192322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:51863)
0620 22:01:18.192343 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:18.192359 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 22:01:18.192497 22244 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:01:18.193149 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 22111
I20260620 22:01:18.193408 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:44437
--local_ip_for_outbound_sockets=127.6.241.131
--tserver_master_addrs=127.6.241.190:32893
--webserver_port=35901
--webserver_interface=127.6.241.131
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:18.272595 22247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:18.272794 22247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:18.272818 22247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:18.273988 22247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:18.274029 22247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:18.274055 22247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:01:18.275507 22247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:44437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=35901
--tserver_master_addrs=127.6.241.190:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:18.275749 22247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:18.275976 22247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:18.280843 22255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:18.280866 22253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:18.280926 22247 server_base.cc:1034] running on GCE node
W20260620 22:01:18.280843 22252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:18.281394 22247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:18.281646 22247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:18.282804 22247 hybrid_clock.cc:648] HybridClock initialized: now 1781992878282789 us; error 26 us; skew 500 ppm
Jun 20 22:01:18 dist-test-slave-gl2g krb5kdc[20811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992878, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:18.285286 22247 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:01:18.285710 22247 webserver.cc:460] Webserver started at http://127.6.241.131:35901/ using document root <none> and password file <none>
I20260620 22:01:18.285928 22247 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:18.285996 22247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:18.287351 22247 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:01:18.287979 22262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:18.288152 22247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:01:18.288255 22247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "19345c3de51a40f7ac34406220433d8b"
format_stamp: "Formatted at 2026-06-20 22:01:16 on dist-test-slave-gl2g"
server_key: "540cf96fb6216095f88cfdf49e33b15c"
server_key_iv: "fe7efaa812e25dcf464df2329ce2ffbd"
server_key_version: "encryptionkey@0"
I20260620 22:01:18.288604 22247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:18.305472 22247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:18.307739 22247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:18.307909 22247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:18.308139 22247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:18.308444 22247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:18.308498 22247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:18.308539 22247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:18.308576 22247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:18.314131 22247 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:44437
I20260620 22:01:18.314206 22375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:44437 every 8 connection(s)
I20260620 22:01:18.314558 22247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:01:18.315614 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:18.315244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:52987)
0620 22:01:18.315402 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:18.315470 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
W20260620 22:01:18.316004 22376 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:32893 (0 consecutive failures): Network error: Failed to ping master at 127.6.241.190:32893: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
I20260620 22:01:18.316278 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:18.316180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:33685)
0620 22:01:18.316230 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:18.316254 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 22:01:18.316552 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:18.316470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:53377)
0620 22:01:18.316512 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:18.316533 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:01:18.316656 22376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:01:18.318002 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 22247
I20260620 22:01:19.128707 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:19.128438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:41873)
0620 22:01:19.128594 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:19.128643 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260620 22:01:19.193009 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:19.192807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:37965)
0620 22:01:19.192918 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:19.192959 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260620 22:01:19.317201 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:19.316971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:45627)
0620 22:01:19.317105 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:19.317144 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260620 22:01:19.785236 22383 raft_consensus.cc:482] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:01:19.785405 22383 raft_consensus.cc:504] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:19.785741 22383 leader_election.cc:288] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c692fd88b81549ce87ec0f34661d3b36 (127.6.241.130:35823), 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437)
Jun 20 22:01:19 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.131@KRBTEST.COM
Jun 20 22:01:19 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.130@KRBTEST.COM
I20260620 22:01:19.796512 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:19.786150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:44437 (local address 127.6.241.129:52099)
0620 22:01:19.786248 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:19.786258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:19.786298 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:19.787281 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:19.787282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:19.787451 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:19.787512 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:19.787514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:19.788544 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:19.788545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:19.789331 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:19.789334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:19.789935 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:19.789936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:19.790044 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:19.790663 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:19.790679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:19.792350 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:19.794961 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:19.794967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:19.794969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:19.795195 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:19.795462 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:19.795463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:19.795463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:19.795572 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:19.795962 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:19.795968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:19.796106 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 22:01:19.796217 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 22:01:19.796382 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":53,"thread_start_us":53,"threads_started":1}
I20260620 22:01:19.796516 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:19.786291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:52099 (local address 127.6.241.131:44437)
0620 22:01:19.786413 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:19.786424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:19.786462 (+ 38us) server_negotiation.cc:407] Connection header received
0620 22:01:19.787090 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:19.787093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:19.787143 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:19.787278 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:19.787615 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:19.788418 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:19.789426 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:19.789831 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:19.789955 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:19.792721 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:19.792726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:19.792726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:19.792738 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:19.794871 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:19.795285 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:19.795288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:19.795288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:19.795404 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:19.795645 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:19.795646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:19.795646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:19.795885 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:19.795980 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:19.796270 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 22:01:19.796390 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
I20260620 22:01:19.796649 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:19.785944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35823 (local address 127.6.241.129:43503)
0620 22:01:19.786080 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:19.786092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:19.786164 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:19.786664 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:19.786667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:19.787053 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:19.787354 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:19.787362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:19.788524 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:19.788525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:19.789330 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:19.789332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:19.789915 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:19.789916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:19.790047 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:19.790750 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:19.790770 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:19.792764 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:19.795414 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:19.795414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:19.795415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:19.795578 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:19.795847 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:19.795848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:19.795848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:19.795905 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:19.796287 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:19.796288 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:19.796334 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 22:01:19.796419 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:01:19.796538 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260620 22:01:19.796757 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:19.786047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:43503 (local address 127.6.241.130:35823)
0620 22:01:19.786255 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:19.786268 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:19.786299 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:01:19.786409 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:19.786410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:19.786451 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:19.786570 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:19.787492 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:19.788411 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:19.789429 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:19.789823 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:19.789939 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:19.792871 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:19.792875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:19.792876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:19.792886 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:19.795338 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:19.795665 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:19.795668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:19.795669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:19.795765 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:19.795980 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:19.795981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:19.795981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:19.796209 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:19.796306 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:19.796504 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 22:01:19.796646 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":61,"threads_started":1}
I20260620 22:01:19.797300 22197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed22d09c9b14fc69e2f4e0040f7d150" candidate_uuid: "4f1f69a40b554ba8be22cffbd4df46df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c692fd88b81549ce87ec0f34661d3b36" is_pre_election: true
I20260620 22:01:19.797458 22197 raft_consensus.cc:2454] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f1f69a40b554ba8be22cffbd4df46df in term 1.
W20260620 22:01:19.797641 21996 leader_election.cc:341] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437): Not found: Tablet not found: bed22d09c9b14fc69e2f4e0040f7d150
I20260620 22:01:19.797726 21996 leader_election.cc:302] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f1f69a40b554ba8be22cffbd4df46df, c692fd88b81549ce87ec0f34661d3b36; no voters: 19345c3de51a40f7ac34406220433d8b
I20260620 22:01:19.797825 22383 raft_consensus.cc:2781] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 22:01:19.797883 22383 raft_consensus.cc:482] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:01:19.797910 22383 raft_consensus.cc:3037] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:01:19.799099 22383 raft_consensus.cc:504] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:19.799249 22383 leader_election.cc:288] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 2 election: Requested vote from peers c692fd88b81549ce87ec0f34661d3b36 (127.6.241.130:35823), 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437)
I20260620 22:01:19.799497 22197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed22d09c9b14fc69e2f4e0040f7d150" candidate_uuid: "4f1f69a40b554ba8be22cffbd4df46df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c692fd88b81549ce87ec0f34661d3b36"
W20260620 22:01:19.799525 21996 leader_election.cc:341] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437): Not found: Tablet not found: bed22d09c9b14fc69e2f4e0040f7d150
I20260620 22:01:19.799589 22197 raft_consensus.cc:3037] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:01:19.800696 22197 raft_consensus.cc:2454] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f1f69a40b554ba8be22cffbd4df46df in term 2.
I20260620 22:01:19.800935 21996 leader_election.cc:302] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f1f69a40b554ba8be22cffbd4df46df, c692fd88b81549ce87ec0f34661d3b36; no voters: 19345c3de51a40f7ac34406220433d8b
I20260620 22:01:19.801023 22383 raft_consensus.cc:2781] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:01:19.801167 22383 raft_consensus.cc:686] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [term 2 LEADER]: Becoming Leader. State: Replica: 4f1f69a40b554ba8be22cffbd4df46df, State: Running, Role: LEADER
I20260620 22:01:19.801261 22383 consensus_queue.cc:237] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:19.801498 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:19.801381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:51013)
0620 22:01:19.801412 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:19.801473 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 22:01:20.193651 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:20.193456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:35765)
0620 22:01:20.193581 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:20.193611 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
W20260620 22:01:20.199473 21996 consensus_peers.cc:487] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df -> Peer 19345c3de51a40f7ac34406220433d8b (127.6.241.131:44437): Couldn't send request to peer 19345c3de51a40f7ac34406220433d8b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bed22d09c9b14fc69e2f4e0040f7d150. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:01:20.200198 22197 raft_consensus.cc:1261] T bed22d09c9b14fc69e2f4e0040f7d150 P c692fd88b81549ce87ec0f34661d3b36 [term 2 FOLLOWER]: Refusing update from remote peer 4f1f69a40b554ba8be22cffbd4df46df: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 22:01:20.200619 22383 consensus_queue.cc:1035] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 22:01:20.201467 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:20.201359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:45497)
0620 22:01:20.201394 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:20.201432 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 22:01:20.202261 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:20.202159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:40545)
0620 22:01:20.202203 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:20.202231 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 22:01:20.202344 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:20.202255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:58039)
0620 22:01:20.202290 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:20.202316 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 22:01:20.317899 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:20.317675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:35341)
0620 22:01:20.317815 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:20.317858 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260620 22:01:20.633886 22396 ts_tablet_manager.cc:927] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Initiating tablet copy from peer 4f1f69a40b554ba8be22cffbd4df46df (127.6.241.129:38673)
I20260620 22:01:20.634164 22396 tablet_copy_client.cc:316] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.241.129:38673
Jun 20 22:01:20 dist-test-slave-gl2g krb5kdc[20811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992878, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.129@KRBTEST.COM
I20260620 22:01:20.641825 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:20.634472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:38673 (local address 127.6.241.131:55487)
0620 22:01:20.634526 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:20.634550 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:20.634654 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:20.634846 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:20.634851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:20.635134 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:20.635204 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:20.635206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:20.636095 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:20.636096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:20.636772 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:20.636775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:20.637201 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:20.637202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:20.637255 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:20.637796 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:20.637801 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:20.639232 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:20.640814 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:20.640820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:20.640821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:20.641008 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:20.641236 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:20.641237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:20.641237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:20.641292 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:20.641543 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:20.641548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:20.641594 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 22:01:20.641661 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 22:01:20.641746 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 22:01:20.641837 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:20.634567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:55487 (local address 127.6.241.129:38673)
0620 22:01:20.634708 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:20.634713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:20.634722 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:01:20.634746 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:20.634748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:20.634760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:20.634835 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:20.635280 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:20.636026 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:20.636841 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:20.637134 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:20.637194 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:20.639306 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:20.639313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:20.639313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:20.639319 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:20.640740 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:20.641102 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:20.641106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:20.641107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:20.641171 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:20.641347 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:20.641348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:20.641348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:20.641489 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:20.641552 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:20.641704 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 22:01:20.641770 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260620 22:01:20.642292 22081 tablet_copy_service.cc:140] P 4f1f69a40b554ba8be22cffbd4df46df: Received BeginTabletCopySession request for tablet bed22d09c9b14fc69e2f4e0040f7d150 from peer 19345c3de51a40f7ac34406220433d8b ({username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:55487)
I20260620 22:01:20.642359 22081 tablet_copy_service.cc:161] P 4f1f69a40b554ba8be22cffbd4df46df: Beginning new tablet copy session on tablet bed22d09c9b14fc69e2f4e0040f7d150 from peer 19345c3de51a40f7ac34406220433d8b at {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:55487: session id = 19345c3de51a40f7ac34406220433d8b-bed22d09c9b14fc69e2f4e0040f7d150
I20260620 22:01:20.642930 22081 tablet_copy_source_session.cc:205] T bed22d09c9b14fc69e2f4e0040f7d150 P 4f1f69a40b554ba8be22cffbd4df46df: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:01:20.643945 22396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed22d09c9b14fc69e2f4e0040f7d150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:20.646693 22396 tablet_copy_client.cc:791] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: tablet copy: Starting download of 0 data blocks...
I20260620 22:01:20.646847 22396 tablet_copy_client.cc:655] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: tablet copy: Starting download of 1 WAL segments...
I20260620 22:01:20.647691 22396 tablet_copy_client.cc:531] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:01:20.648329 22396 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Bootstrap starting.
I20260620 22:01:20.649560 22396 log.cc:824] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:20.651870 22396 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:20.652045 22396 tablet_bootstrap.cc:492] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Bootstrap complete.
I20260620 22:01:20.652137 22396 ts_tablet_manager.cc:1397] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 22:01:20.652979 22396 raft_consensus.cc:348] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:20.653115 22396 raft_consensus.cc:729] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19345c3de51a40f7ac34406220433d8b, State: Initialized, Role: FOLLOWER
I20260620 22:01:20.653235 22396 consensus_queue.cc:260] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c692fd88b81549ce87ec0f34661d3b36" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35823 } } peers { permanent_uuid: "19345c3de51a40f7ac34406220433d8b" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 44437 } } peers { permanent_uuid: "4f1f69a40b554ba8be22cffbd4df46df" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38673 } }
I20260620 22:01:20.653505 22396 ts_tablet_manager.cc:1428] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260620 22:01:20.653823 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:20.653708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:43619)
0620 22:01:20.653762 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:20.653790 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 22:01:20.653806 22081 tablet_copy_service.cc:342] P 4f1f69a40b554ba8be22cffbd4df46df: Request end of tablet copy session 19345c3de51a40f7ac34406220433d8b-bed22d09c9b14fc69e2f4e0040f7d150 received from {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:55487
I20260620 22:01:20.653872 22081 tablet_copy_service.cc:433] P 4f1f69a40b554ba8be22cffbd4df46df: ending tablet copy session 19345c3de51a40f7ac34406220433d8b-bed22d09c9b14fc69e2f4e0040f7d150 on tablet bed22d09c9b14fc69e2f4e0040f7d150 with peer 19345c3de51a40f7ac34406220433d8b
I20260620 22:01:20.654209 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:20.654086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:37417)
0620 22:01:20.654137 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:20.654166 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:01:21.031796 22330 raft_consensus.cc:1203] T bed22d09c9b14fc69e2f4e0040f7d150 P 19345c3de51a40f7ac34406220433d8b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 22:01:21.031874 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:21.031706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:53669)
0620 22:01:21.031802 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:21.031838 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":42,"threads_started":1}
I20260620 22:01:21.032399 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:21.032236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:59937)
0620 22:01:21.032317 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:21.032360 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 22:01:21.202948 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:21.202746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:52567)
0620 22:01:21.202853 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:21.202902 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260620 22:01:22.032557 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:22.032304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:51505)
0620 22:01:22.032444 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:22.032493 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260620 22:01:22.033119 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:22.032884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:57103)
0620 22:01:22.033032 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:22.033081 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":59,"threads_started":1}
I20260620 22:01:22.203572 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:22.203362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:47973)
0620 22:01:22.203485 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:22.203523 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260620 22:01:23.033290 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:23.033089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:38475)
0620 22:01:23.033212 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:23.033249 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260620 22:01:23.033672 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:23.033580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:37045)
0620 22:01:23.033631 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:23.033655 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:01:23.033819 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:23.033579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:33103)
0620 22:01:23.033727 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:23.033767 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260620 22:01:23.204365 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:23.204166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:41081)
0620 22:01:23.204285 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:23.204323 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260620 22:01:24.034302 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:24.034068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:46523)
0620 22:01:24.034207 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:24.034258 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260620 22:01:24.034574 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:24.034332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:33875)
0620 22:01:24.034509 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:24.034536 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":66,"threads_started":1}
I20260620 22:01:24.205037 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:24.204830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:45379)
0620 22:01:24.204957 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:24.204992 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260620 22:01:25.035058 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:25.034776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:60567)
0620 22:01:25.034950 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:25.034982 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
I20260620 22:01:25.035221 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:25.034979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:41675)
0620 22:01:25.035117 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:25.035169 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260620 22:01:25.205696 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:25.205452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:53641)
0620 22:01:25.205610 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:25.205652 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
I20260620 22:01:26.035725 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:26.035476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:39451)
0620 22:01:26.035618 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:26.035677 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260620 22:01:26.035813 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:26.035617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:41733)
0620 22:01:26.035743 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:26.035776 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260620 22:01:26.206363 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:26.206110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:44925)
0620 22:01:26.206284 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:26.206320 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260620 22:01:27.036491 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:27.036233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:44927)
0620 22:01:27.036388 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:27.036425 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260620 22:01:27.036518 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:27.036335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:45393)
0620 22:01:27.036440 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:27.036469 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260620 22:01:27.207111 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:27.206859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:47215)
0620 22:01:27.207022 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:27.207061 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260620 22:01:28.037154 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:28.036945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:54045)
0620 22:01:28.037081 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:28.037115 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260620 22:01:28.037155 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:28.036936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:56541)
0620 22:01:28.037080 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:28.037114 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260620 22:01:28.207845 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:28.207616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:39065)
0620 22:01:28.207753 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:28.207797 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260620 22:01:29.037778 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:29.037578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:60271)
0620 22:01:29.037708 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:29.037740 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":85,"threads_started":1}
I20260620 22:01:29.037786 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:29.037590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:50159)
0620 22:01:29.037712 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:29.037745 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260620 22:01:29.208454 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:29.208247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:58195)
0620 22:01:29.208376 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:29.208412 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260620 22:01:30.038513 22428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:30.038287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:44191)
0620 22:01:30.038445 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:30.038475 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260620 22:01:30.038511 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:30.038287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:41725)
0620 22:01:30.038422 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:30.038468 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260620 22:01:30.209089 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:30.208887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:58641)
0620 22:01:30.209016 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:30.209047 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260620 22:01:31.039207 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:31.038985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:46567)
0620 22:01:31.039122 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:31.039155 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260620 22:01:31.039206 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:31.039012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:35117)
0620 22:01:31.039128 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:31.039166 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260620 22:01:31.209760 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:31.209538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:37361)
0620 22:01:31.209677 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:31.209715 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260620 22:01:32.039914 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:32.039691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:35299)
0620 22:01:32.039809 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:32.039845 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":114,"threads_started":1}
I20260620 22:01:32.039965 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:32.039720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:44123)
0620 22:01:32.039898 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:32.039927 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260620 22:01:32.210357 22436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:32.210136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:34247)
0620 22:01:32.210279 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:32.210312 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260620 22:01:33.040728 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:33.040534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.129:44509)
0620 22:01:33.040655 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:33.040696 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260620 22:01:33.040784 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:33.040592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.131:41535)
0620 22:01:33.040717 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:33.040746 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260620 22:01:33.211130 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:33.210943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:32893 (local address 127.6.241.130:51223)
0620 22:01:33.211059 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:33.211089 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260620 22:01:33.327291 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 21948
I20260620 22:01:33.332036 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 22111
I20260620 22:01:33.336310 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 22247
2026-06-20T22:01:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18523 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:01:33 dist-test-slave-gl2g krb5kdc[22442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:01:33 dist-test-slave-gl2g krb5kdc[22442](info): set up 2 sockets
Jun 20 22:01:33 dist-test-slave-gl2g krb5kdc[22442](info): commencing operation
krb5kdc: starting...
W20260620 22:01:34.366828 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:01:34 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:01:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:01:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:01:34.474970 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41935
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:41935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:34.553433 22455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:34.553629 22455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:34.553651 22455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:34.554811 22455 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:01:34.554849 22455 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:01:34.554863 22455 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:01:34.554874 22455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:34.554888 22455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:34.554900 22455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:34.554952 22455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:34.554968 22455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:34.556458 22455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:41935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:34.556730 22455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:34.556962 22455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:34.561579 22463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:34.561564 22460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:34.561571 22461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:34.561780 22455 server_base.cc:1034] running on GCE node
I20260620 22:01:34.561981 22455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:34.562239 22455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:34.563367 22455 hybrid_clock.cc:648] HybridClock initialized: now 1781992894563355 us; error 27 us; skew 500 ppm
Jun 20 22:01:34 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992894, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:34.565663 22455 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:01:34.566087 22455 webserver.cc:460] Webserver started at http://127.6.241.190:43623/ using document root <none> and password file <none>
I20260620 22:01:34.566305 22455 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:34.566372 22455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:34.566515 22455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:34.567574 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "e0d1cf518d8e4cd997670173b330c4a3"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "a1cca8f62efd328667757cc8c2134c7d"
server_key_iv: "e0427571b22e5617a60df2c55136f50d"
server_key_version: "encryptionkey@0"
I20260620 22:01:34.567937 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "e0d1cf518d8e4cd997670173b330c4a3"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "a1cca8f62efd328667757cc8c2134c7d"
server_key_iv: "e0427571b22e5617a60df2c55136f50d"
server_key_version: "encryptionkey@0"
I20260620 22:01:34.569269 22455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:01:34.569927 22470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:34.570142 22455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:01:34.570238 22455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "e0d1cf518d8e4cd997670173b330c4a3"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "a1cca8f62efd328667757cc8c2134c7d"
server_key_iv: "e0427571b22e5617a60df2c55136f50d"
server_key_version: "encryptionkey@0"
I20260620 22:01:34.570317 22455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:34.586383 22455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:34.588191 22455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:34.588343 22455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:34.592121 22455 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:41935
I20260620 22:01:34.592195 22522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:41935 every 8 connection(s)
I20260620 22:01:34.592532 22455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:01:34.593326 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:34.595548 22523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Bootstrap starting.
I20260620 22:01:34.596072 22523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:34.596303 22523 log.cc:824] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:34.596907 22523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: No bootstrap required, opened a new log
I20260620 22:01:34.598351 22523 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:01:34.598479 22523 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:34.598500 22523 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d1cf518d8e4cd997670173b330c4a3, State: Initialized, Role: FOLLOWER
I20260620 22:01:34.598654 22523 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:01:34.598734 22523 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:01:34.598757 22523 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:01:34.598785 22523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:34.598948 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 22455
I20260620 22:01:34.599056 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:01:34.599228 7110 external_mini_cluster.cc:1428] Setting key 8be682dc04d718ac4d5f56e2e8396657
I20260620 22:01:34.599442 22523 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:01:34.599526 22523 leader_election.cc:302] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d1cf518d8e4cd997670173b330c4a3; no voters:
I20260620 22:01:34.599713 22523 leader_election.cc:288] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:01:34.599910 22528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:34.599979 22523 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:34.600334 22528 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 1 LEADER]: Becoming Leader. State: Replica: e0d1cf518d8e4cd997670173b330c4a3, State: Running, Role: LEADER
I20260620 22:01:34.600430 22528 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:01:34.600760 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0d1cf518d8e4cd997670173b330c4a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } } }
I20260620 22:01:34.600835 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:34.600962 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d1cf518d8e4cd997670173b330c4a3. Latest consensus state: current_term: 1 leader_uuid: "e0d1cf518d8e4cd997670173b330c4a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } } }
I20260620 22:01:34.601017 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:34.601226 22537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:01:34.601951 22537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:01:34.606742 22537 catalog_manager.cc:1347] Generated new cluster ID: 43fad989ceb5452aab30b17574988905
I20260620 22:01:34.606797 22537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:01:34 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:34.611326 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:34.600035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.6.241.190:41935)
0620 22:01:34.600279 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:34.600288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:34.600319 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:01:34.600436 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:34.600443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:34.600508 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:34.600674 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:34.601513 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.602893 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.604038 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.604660 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.604778 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:34.607619 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:34.607623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:34.607623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:34.607634 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:34.609745 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:34.610127 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:34.610130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:34.610130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:34.610221 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:34.610426 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:34.610427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:34.610427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:34.610734 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:34.610866 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:34.610979 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 22:01:34.611118 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260620 22:01:34.620071 22537 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:01:34.620750 22537 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:01:34.626080 22537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Generated new TSK 0
I20260620 22:01:34.626237 22537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 22:01:34.649433 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41935
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:34.727133 22549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:34.727357 22549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:34.727411 22549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:34.728523 22549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:34.728601 22549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:34.728658 22549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:01:34.730067 22549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:34.730319 22549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:34.730535 22549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:34.735306 22557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:34.735415 22554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:34.735291 22555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:34.735500 22549 server_base.cc:1034] running on GCE node
I20260620 22:01:34.735721 22549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:34.736021 22549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:34.737178 22549 hybrid_clock.cc:648] HybridClock initialized: now 1781992894737173 us; error 36 us; skew 500 ppm
Jun 20 22:01:34 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992894, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:34.739673 22549 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:01:34.740120 22549 webserver.cc:460] Webserver started at http://127.6.241.129:41969/ using document root <none> and password file <none>
I20260620 22:01:34.740347 22549 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:34.740391 22549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:34.740548 22549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:34.741510 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "250124f3d4184342b7dafbb0b48831c7"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "8dc390df67b0a5016475542461939eb3"
server_key_iv: "e2d744939b33930626c0e3fff1705e34"
server_key_version: "encryptionkey@0"
I20260620 22:01:34.741852 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "250124f3d4184342b7dafbb0b48831c7"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "8dc390df67b0a5016475542461939eb3"
server_key_iv: "e2d744939b33930626c0e3fff1705e34"
server_key_version: "encryptionkey@0"
I20260620 22:01:34.743232 22549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:01:34.743964 22564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:34.744171 22549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:34.744240 22549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "250124f3d4184342b7dafbb0b48831c7"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "8dc390df67b0a5016475542461939eb3"
server_key_iv: "e2d744939b33930626c0e3fff1705e34"
server_key_version: "encryptionkey@0"
I20260620 22:01:34.744333 22549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:34.767645 22549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:34.769838 22549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:34.769958 22549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:34.770176 22549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:34.770467 22549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:34.770500 22549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:34.770521 22549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:34.770568 22549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:34.776448 22549 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:42537
I20260620 22:01:34.776495 22677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:42537 every 8 connection(s)
I20260620 22:01:34.776794 22549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:01:34 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992894, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:34.783601 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 22549
I20260620 22:01:34.783707 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:01:34.783900 7110 external_mini_cluster.cc:1428] Setting key a7e9baf54d9a8f2b4e5f7e0e4bb9b499
I20260620 22:01:34.785375 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:34.777361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:41659 (local address 127.6.241.190:41935)
0620 22:01:34.777410 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:34.777413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:34.777642 (+ 229us) server_negotiation.cc:407] Connection header received
0620 22:01:34.778081 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:34.778081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:34.778091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:34.778143 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:34.779057 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.779543 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.780461 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.780659 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.780707 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:34.782721 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:34.782723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:34.782723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:34.782729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:34.783994 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:34.784423 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:34.784424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:34.784425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:34.784484 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:34.784763 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:34.784764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:34.784764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:34.784860 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:34.784924 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:34.785253 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 22:01:34.785316 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:01:34.785656 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:34.777367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:41659)
0620 22:01:34.777533 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:34.777567 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:34.778004 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:34.778206 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:34.778209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:34.778808 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:34.778986 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:34.778991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.779619 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.779620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:34.780395 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:34.780397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.780733 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.780733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:34.780827 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:34.781313 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:34.781322 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:34.782581 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:34.784074 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:34.784080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:34.784082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:34.784356 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:34.784554 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:34.784554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:34.784555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:34.784685 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:34.784936 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:34.784942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:34.785118 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 22:01:34.785234 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 22:01:34.785397 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260620 22:01:34.786208 22678 heartbeater.cc:347] Connected to a master server at 127.6.241.190:41935
I20260620 22:01:34.786320 22678 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:34.786484 22678 heartbeater.cc:511] Master 127.6.241.190:41935 requested a full tablet report, sending...
I20260620 22:01:34.787032 22487 ts_manager.cc:194] Registered new tserver with Master: 250124f3d4184342b7dafbb0b48831c7 (127.6.241.129:42537)
I20260620 22:01:34.787983 22487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:41659
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 22:01:34.807039 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41935
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:34.885726 22683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:34.885905 22683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:34.885926 22683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:34.887035 22683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:34.887074 22683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:34.887100 22683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:01:34.888418 22683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:34.888579 22683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:34.888731 22683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:34.893384 22691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:34.893440 22683 server_base.cc:1034] running on GCE node
W20260620 22:01:34.893384 22688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:34.893373 22689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:34.893814 22683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:34.894037 22683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:34.895190 22683 hybrid_clock.cc:648] HybridClock initialized: now 1781992894895182 us; error 42 us; skew 500 ppm
Jun 20 22:01:34 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992894, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:34.897590 22683 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:01:34.897989 22683 webserver.cc:460] Webserver started at http://127.6.241.130:44471/ using document root <none> and password file <none>
I20260620 22:01:34.898205 22683 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:34.898273 22683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:34.898416 22683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:34.899448 22683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "63c905d15cd946ba901971361636cec9"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "f2b7d4c48784f4821aa304a8ef0880a4"
server_key_iv: "082db7d9d10695576fe957a72501c3a9"
server_key_version: "encryptionkey@0"
I20260620 22:01:34.899791 22683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "63c905d15cd946ba901971361636cec9"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "f2b7d4c48784f4821aa304a8ef0880a4"
server_key_iv: "082db7d9d10695576fe957a72501c3a9"
server_key_version: "encryptionkey@0"
I20260620 22:01:34.901140 22683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:01:34.901822 22698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:34.901989 22683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:01:34.902083 22683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "63c905d15cd946ba901971361636cec9"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "f2b7d4c48784f4821aa304a8ef0880a4"
server_key_iv: "082db7d9d10695576fe957a72501c3a9"
server_key_version: "encryptionkey@0"
I20260620 22:01:34.902163 22683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:34.923900 22683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:34.926045 22683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:34.926196 22683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:34.926409 22683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:34.926726 22683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:34.926781 22683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:34.926826 22683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:34.926853 22683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:34.932521 22683 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:34045
I20260620 22:01:34.932600 22811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:34045 every 8 connection(s)
I20260620 22:01:34.932894 22683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:01:34 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992894, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:34.941326 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:34.933536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:50397 (local address 127.6.241.190:41935)
0620 22:01:34.933603 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:34.933605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:34.933798 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:01:34.934269 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:34.934270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:34.934280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:34.934335 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:34.935236 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.935724 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.936655 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.936829 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.936877 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:34.938873 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:34.938874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:34.938875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:34.938880 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:34.940273 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:34.940612 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:34.940612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:34.940613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:34.940661 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:34.940864 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:34.940865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:34.940865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:34.940961 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:34.941010 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:34.941198 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 22:01:34.941265 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:01:34.941494 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 22683
I20260620 22:01:34.941555 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:34.933496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:50397)
0620 22:01:34.933664 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:34.933701 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:34.934182 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:34.934385 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:34.934387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:34.934997 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:34.935166 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:34.935170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.935808 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.935809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:34.936579 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:34.936581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:34.936895 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:34.936896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:34.936992 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:34.937460 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:34.937468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:34.938794 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:34.940352 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:34.940357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:34.940358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:34.940552 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:34.940717 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:34.940718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:34.940718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:34.940804 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:34.941013 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:34.941016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:34.941114 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 22:01:34.941198 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:01:34.941303 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260620 22:01:34.941587 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:01:34.941834 7110 external_mini_cluster.cc:1428] Setting key d89dfeeeadaedea830892e82c522aa8e
I20260620 22:01:34.942003 22812 heartbeater.cc:347] Connected to a master server at 127.6.241.190:41935
I20260620 22:01:34.942097 22812 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:34.942256 22812 heartbeater.cc:511] Master 127.6.241.190:41935 requested a full tablet report, sending...
I20260620 22:01:34.942674 22487 ts_manager.cc:194] Registered new tserver with Master: 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045)
I20260620 22:01:34.943051 22487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:50397
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 22:01:34.964401 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41935
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:35.042492 22817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:35.042693 22817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:35.042716 22817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:35.043769 22817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:35.043800 22817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:35.043824 22817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:01:35.045174 22817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:35.045351 22817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:35.045521 22817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:35.050452 22825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:35.050452 22823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:35.050474 22822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:35.050592 22817 server_base.cc:1034] running on GCE node
I20260620 22:01:35.051015 22817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:35.051215 22817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:35.052351 22817 hybrid_clock.cc:648] HybridClock initialized: now 1781992895052338 us; error 30 us; skew 500 ppm
Jun 20 22:01:35 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992895, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:35.054862 22817 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:01:35.055373 22817 webserver.cc:460] Webserver started at http://127.6.241.131:36681/ using document root <none> and password file <none>
I20260620 22:01:35.055601 22817 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:35.055645 22817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:35.055806 22817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:01:35.056846 22817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "eeb4c1a2b64847d284ab66ffd2c457d1"
format_stamp: "Formatted at 2026-06-20 22:01:35 on dist-test-slave-gl2g"
server_key: "35d5e6eecf49cb09e82fdf88d7a9a9b5"
server_key_iv: "71da8ee5d1097ea9e1dfeaa332c13595"
server_key_version: "encryptionkey@0"
I20260620 22:01:35.057274 22817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "eeb4c1a2b64847d284ab66ffd2c457d1"
format_stamp: "Formatted at 2026-06-20 22:01:35 on dist-test-slave-gl2g"
server_key: "35d5e6eecf49cb09e82fdf88d7a9a9b5"
server_key_iv: "71da8ee5d1097ea9e1dfeaa332c13595"
server_key_version: "encryptionkey@0"
I20260620 22:01:35.058589 22817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:01:35.059360 22832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:35.059543 22817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:01:35.059629 22817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "eeb4c1a2b64847d284ab66ffd2c457d1"
format_stamp: "Formatted at 2026-06-20 22:01:35 on dist-test-slave-gl2g"
server_key: "35d5e6eecf49cb09e82fdf88d7a9a9b5"
server_key_iv: "71da8ee5d1097ea9e1dfeaa332c13595"
server_key_version: "encryptionkey@0"
I20260620 22:01:35.059703 22817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:35.068487 22817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:35.070711 22817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:35.070865 22817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:35.071075 22817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:35.071369 22817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:35.071406 22817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:35.071427 22817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:35.071472 22817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:35.077344 22817 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:36041
I20260620 22:01:35.077385 22945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:36041 every 8 connection(s)
I20260620 22:01:35.077668 22817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:01:35.078495 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 22817
I20260620 22:01:35.078641 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:01:35.078840 7110 external_mini_cluster.cc:1428] Setting key 1fffccc4e563e123c205f5a2fd83839f
Jun 20 22:01:35 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992895, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:35.086288 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.078345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:55327 (local address 127.6.241.190:41935)
0620 22:01:35.078394 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:35.078396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:35.078643 (+ 247us) server_negotiation.cc:407] Connection header received
0620 22:01:35.079281 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:35.079282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:35.079292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:35.079365 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:35.080325 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.080804 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.081686 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.081885 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.081935 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.083864 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:35.083865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:35.083866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:35.083872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:35.085238 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:35.085558 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:35.085559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:35.085559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:35.085608 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:35.085807 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:35.085808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:35.085808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:35.085903 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:35.085951 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:35.086163 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 22:01:35.086227 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:01:35.086445 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.078335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:55327)
0620 22:01:35.078506 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:35.078540 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:35.079172 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:35.079437 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:35.079442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:35.080066 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:35.080243 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.080248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.080876 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.080877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.081616 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.081617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.081956 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.081956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.082056 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.082437 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:35.082445 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:35.083782 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:35.085303 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:35.085308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:35.085309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:35.085498 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:35.085662 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:35.085663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:35.085663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:35.085754 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:35.085956 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:35.085959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:35.086059 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 22:01:35.086135 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:01:35.086240 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260620 22:01:35.086941 22946 heartbeater.cc:347] Connected to a master server at 127.6.241.190:41935
I20260620 22:01:35.087038 22946 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:35.087200 22946 heartbeater.cc:511] Master 127.6.241.190:41935 requested a full tablet report, sending...
I20260620 22:01:35.087539 22487 ts_manager.cc:194] Registered new tserver with Master: eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041)
I20260620 22:01:35.087903 22487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:55327
I20260620 22:01:35.090585 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:01:35.097947 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.091909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.6.241.190:41935)
0620 22:01:35.091978 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:35.091980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:35.092096 (+ 116us) server_negotiation.cc:407] Connection header received
0620 22:01:35.092169 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:35.092170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:35.092184 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:35.092248 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:35.092788 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.093237 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.093957 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.094138 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.094215 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.095391 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:35.095392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:35.095393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:35.095399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:35.096777 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:35.097119 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:35.097120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:35.097121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:35.097177 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:35.097374 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:35.097375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:35.097376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:35.097513 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:35.097614 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:35.097771 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 22:01:35.097848 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:01:35.099500 22487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:01:35.101075 22487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:01:35.107098 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.103160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:36041 (local address 127.0.0.1:47262)
0620 22:01:35.103704 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:35.103713 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:35.103770 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:35.103997 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:35.103998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:35.104000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:35.104048 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.104050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.105044 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.105045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.106055 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.106058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.106766 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.106767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.106848 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.106861 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:01:35.106969 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 22:01:35.107029 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":66,"thread_start_us":60,"threads_started":1}
I20260620 22:01:35.107115 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.103354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.6.241.131:36041)
0620 22:01:35.103533 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:35.103537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:35.103769 (+ 232us) server_negotiation.cc:407] Connection header received
0620 22:01:35.103835 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:35.103838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:35.103854 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:35.103930 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:35.104121 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.104948 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.106141 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.106666 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.106764 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.106822 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:35.107005 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 22:01:35.107044 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260620 22:01:35.107544 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.103151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42537 (local address 127.0.0.1:39464)
0620 22:01:35.103311 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:35.103348 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:35.103413 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:35.103713 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:35.103717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:35.103720 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:35.103780 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.103782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.105105 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.105106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.106287 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.106290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.107313 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.107314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.107383 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.107386 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:01:35.107437 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:01:35.107476 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":13,"thread_start_us":61,"threads_started":1}
I20260620 22:01:35.107944 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.103164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:34045 (local address 127.0.0.1:51858)
0620 22:01:35.103850 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:35.103858 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:35.103898 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:35.104152 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:35.104152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:35.104154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:35.104185 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.104187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.105427 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.105428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.106761 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.106763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.107727 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.107728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.107783 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.107786 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:01:35.107854 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:01:35.107887 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":55,"threads_started":1}
I20260620 22:01:35.107992 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.103523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.6.241.130:34045)
0620 22:01:35.103680 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:35.103686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:35.103954 (+ 268us) server_negotiation.cc:407] Connection header received
0620 22:01:35.103996 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:35.103998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:35.104012 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:35.104078 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:35.104270 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.105324 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.106848 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.107565 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.107630 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.107687 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:35.107869 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 22:01:35.107909 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260620 22:01:35.107926 22880 tablet_service.cc:1467] Processing CreateTablet for tablet 082d73956d63480bac43b1d44ad65438 (DEFAULT_TABLE table=test-table [id=bb16a332a4e5488fbcb594d725b7664b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:35.108280 22880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082d73956d63480bac43b1d44ad65438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:35.108634 22746 tablet_service.cc:1467] Processing CreateTablet for tablet 082d73956d63480bac43b1d44ad65438 (DEFAULT_TABLE table=test-table [id=bb16a332a4e5488fbcb594d725b7664b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:35.108933 22746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082d73956d63480bac43b1d44ad65438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:35.110807 22960 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: Bootstrap starting.
I20260620 22:01:35.110754 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.103338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.6.241.129:42537)
0620 22:01:35.103525 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:35.103531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:35.103542 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:01:35.103582 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:35.103583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:35.103598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:35.103703 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:35.103842 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.104992 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.106375 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.107157 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.107233 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.107279 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:35.107465 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 22:01:35.110667 (+ 3202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260620 22:01:35.111682 22960 tablet_bootstrap.cc:654] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:35.111720 22961 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Bootstrap starting.
I20260620 22:01:35.111732 22612 tablet_service.cc:1467] Processing CreateTablet for tablet 082d73956d63480bac43b1d44ad65438 (DEFAULT_TABLE table=test-table [id=bb16a332a4e5488fbcb594d725b7664b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:35.112056 22612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082d73956d63480bac43b1d44ad65438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:35.112069 22960 log.cc:824] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:35.112763 22961 tablet_bootstrap.cc:654] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:35.113687 22961 log.cc:824] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:35.114500 22960 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: No bootstrap required, opened a new log
I20260620 22:01:35.114544 22960 ts_tablet_manager.cc:1397] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 22:01:35.116367 22960 raft_consensus.cc:348] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:35.116534 22960 raft_consensus.cc:374] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:35.116613 22960 raft_consensus.cc:729] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb4c1a2b64847d284ab66ffd2c457d1, State: Initialized, Role: FOLLOWER
I20260620 22:01:35.116736 22960 consensus_queue.cc:260] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:35.117017 22946 heartbeater.cc:503] Master 127.6.241.190:41935 was elected leader, sending a full tablet report...
I20260620 22:01:35.117014 22960 ts_tablet_manager.cc:1428] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 22:01:35.117908 22961 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: No bootstrap required, opened a new log
I20260620 22:01:35.117991 22961 ts_tablet_manager.cc:1397] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 22:01:35.119693 22961 raft_consensus.cc:348] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:35.119853 22961 raft_consensus.cc:374] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:35.119899 22961 raft_consensus.cc:729] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c905d15cd946ba901971361636cec9, State: Initialized, Role: FOLLOWER
I20260620 22:01:35.120003 22961 consensus_queue.cc:260] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:35.120273 22961 ts_tablet_manager.cc:1428] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:01:35.120375 22965 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Bootstrap starting.
I20260620 22:01:35.120424 22812 heartbeater.cc:503] Master 127.6.241.190:41935 was elected leader, sending a full tablet report...
I20260620 22:01:35.121057 22965 tablet_bootstrap.cc:654] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:35.121606 22965 log.cc:824] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:35.122349 22965 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: No bootstrap required, opened a new log
I20260620 22:01:35.122419 22965 ts_tablet_manager.cc:1397] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:01:35.124895 22965 raft_consensus.cc:348] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:35.125087 22965 raft_consensus.cc:374] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:35.125123 22965 raft_consensus.cc:729] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250124f3d4184342b7dafbb0b48831c7, State: Initialized, Role: FOLLOWER
I20260620 22:01:35.125240 22965 consensus_queue.cc:260] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:35.125516 22965 ts_tablet_manager.cc:1428] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:01:35.125535 22678 heartbeater.cc:503] Master 127.6.241.190:41935 was elected leader, sending a full tablet report...
I20260620 22:01:35.183137 22966 raft_consensus.cc:482] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:01:35.183288 22966 raft_consensus.cc:504] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
W20260620 22:01:35.183476 22813 tablet.cc:2369] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:01:35.183701 22966 leader_election.cc:288] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 250124f3d4184342b7dafbb0b48831c7 (127.6.241.129:42537), eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041)
I20260620 22:01:35.187372 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.183845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42537 (local address 127.6.241.130:39171)
0620 22:01:35.183898 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:35.183912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:35.183957 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:35.184189 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:35.184190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:35.184192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:35.184292 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.184295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.185351 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.185352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.186391 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.186392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.187050 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.187051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.187143 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.187146 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:01:35.187219 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:01:35.187267 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:01:35.187493 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.184091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:34001 (local address 127.6.241.131:36041)
0620 22:01:35.184136 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:35.184138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:35.184147 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:01:35.184176 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:35.184177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:35.184187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:35.184252 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:35.184448 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.184924 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.186341 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.187121 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.187222 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.187232 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:35.187364 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 22:01:35.187417 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:01:35.187733 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.183873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:36041 (local address 127.6.241.130:34001)
0620 22:01:35.184052 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:35.184062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:35.184103 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:35.184272 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:35.184273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:35.184275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:35.184329 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.184331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.185013 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.185014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.186245 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:35.186248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.187209 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.187209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:35.187275 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.187279 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:01:35.187604 (+ 325us) client_negotiation.cc:240] Negotiation successful
0620 22:01:35.187656 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":104,"threads_started":1}
I20260620 22:01:35.187930 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.183985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:39171 (local address 127.6.241.129:42537)
0620 22:01:35.184045 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:35.184048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:35.184055 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:01:35.184083 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:35.184084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:35.184093 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:35.184140 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:35.184386 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.185261 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.186476 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.186941 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.187045 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.187054 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:35.187816 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 22:01:35.187866 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:01:35.188146 22900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082d73956d63480bac43b1d44ad65438" candidate_uuid: "63c905d15cd946ba901971361636cec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" is_pre_election: true
I20260620 22:01:35.188163 22632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082d73956d63480bac43b1d44ad65438" candidate_uuid: "63c905d15cd946ba901971361636cec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250124f3d4184342b7dafbb0b48831c7" is_pre_election: true
I20260620 22:01:35.188310 22632 raft_consensus.cc:2454] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c905d15cd946ba901971361636cec9 in term 0.
I20260620 22:01:35.188310 22900 raft_consensus.cc:2454] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c905d15cd946ba901971361636cec9 in term 0.
I20260620 22:01:35.188556 22701 leader_election.cc:302] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250124f3d4184342b7dafbb0b48831c7, 63c905d15cd946ba901971361636cec9; no voters:
I20260620 22:01:35.188746 22966 raft_consensus.cc:2781] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:01:35.188838 22966 raft_consensus.cc:482] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:01:35.188871 22966 raft_consensus.cc:3037] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:35.189559 22966 raft_consensus.cc:504] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:35.189700 22966 leader_election.cc:288] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [CANDIDATE]: Term 1 election: Requested vote from peers 250124f3d4184342b7dafbb0b48831c7 (127.6.241.129:42537), eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041)
I20260620 22:01:35.189843 22632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082d73956d63480bac43b1d44ad65438" candidate_uuid: "63c905d15cd946ba901971361636cec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250124f3d4184342b7dafbb0b48831c7"
I20260620 22:01:35.189862 22900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082d73956d63480bac43b1d44ad65438" candidate_uuid: "63c905d15cd946ba901971361636cec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1"
I20260620 22:01:35.189939 22632 raft_consensus.cc:3037] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:35.189952 22900 raft_consensus.cc:3037] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:35.190760 22632 raft_consensus.cc:2454] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c905d15cd946ba901971361636cec9 in term 1.
I20260620 22:01:35.190896 22701 leader_election.cc:302] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250124f3d4184342b7dafbb0b48831c7, 63c905d15cd946ba901971361636cec9; no voters:
I20260620 22:01:35.190974 22966 raft_consensus.cc:2781] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:35.191083 22900 raft_consensus.cc:2454] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c905d15cd946ba901971361636cec9 in term 1.
I20260620 22:01:35.191128 22966 raft_consensus.cc:686] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 1 LEADER]: Becoming Leader. State: Replica: 63c905d15cd946ba901971361636cec9, State: Running, Role: LEADER
I20260620 22:01:35.191229 22966 consensus_queue.cc:237] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:35.191958 22487 catalog_manager.cc:5495] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c905d15cd946ba901971361636cec9 (127.6.241.130). New cstate: current_term: 1 leader_uuid: "63c905d15cd946ba901971361636cec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } health_report { overall_health: UNKNOWN } } }
I20260620 22:01:35.210134 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.207804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.6.241.130:34045)
0620 22:01:35.207854 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:35.207856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:35.207865 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:01:35.207922 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:35.207923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:35.207946 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:35.208021 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:01:35.208159 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.208616 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.209351 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:35.209536 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:35.209609 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:35.209802 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:01:35.209895 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:01:35.209988 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:35.210013 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 22:01:35.210032 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:01:35.211772 22632 raft_consensus.cc:1261] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Refusing update from remote peer 63c905d15cd946ba901971361636cec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:35.211833 22900 raft_consensus.cc:1261] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [term 1 FOLLOWER]: Refusing update from remote peer 63c905d15cd946ba901971361636cec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:35.212154 22970 consensus_queue.cc:1035] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:35.212306 22966 consensus_queue.cc:1035] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:35.214946 22974 mvcc.cc:205] Tried to move back new op lower bound from 7299042898785120256 to 7299042898703745024. Current Snapshot: MvccSnapshot[applied={T|T < 7299042898785120256}]
I20260620 22:01:35.215034 22973 mvcc.cc:205] Tried to move back new op lower bound from 7299042898785120256 to 7299042898703745024. Current Snapshot: MvccSnapshot[applied={T|T < 7299042898785120256}]
I20260620 22:01:35.217605 22486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:35.217705 22486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:35.218441 22486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Sending DeleteTablet for 3 replicas of tablet 082d73956d63480bac43b1d44ad65438
I20260620 22:01:35.218808 22612 tablet_service.cc:1514] Processing DeleteTablet for tablet 082d73956d63480bac43b1d44ad65438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:35 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:39464
I20260620 22:01:35.218806 22880 tablet_service.cc:1514] Processing DeleteTablet for tablet 082d73956d63480bac43b1d44ad65438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:35 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:47262
I20260620 22:01:35.218808 22746 tablet_service.cc:1514] Processing DeleteTablet for tablet 082d73956d63480bac43b1d44ad65438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:35 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:51858
I20260620 22:01:35.219089 22985 tablet_replica.cc:330] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: stopping tablet replica
I20260620 22:01:35.219096 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 22549
I20260620 22:01:35.219125 22984 tablet_replica.cc:330] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: stopping tablet replica
I20260620 22:01:35.219853 22985 raft_consensus.cc:2229] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:35.220018 22985 raft_consensus.cc:2258] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:35.220431 22985 ts_tablet_manager.cc:1905] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:35.221866 22985 ts_tablet_manager.cc:1918] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:35.221946 22985 log.cc:1196] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/082d73956d63480bac43b1d44ad65438
I20260620 22:01:35.222201 22985 ts_tablet_manager.cc:1939] T 082d73956d63480bac43b1d44ad65438 P eeb4c1a2b64847d284ab66ffd2c457d1: Deleting consensus metadata
I20260620 22:01:35.222584 22474 catalog_manager.cc:4841] TS eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041): tablet 082d73956d63480bac43b1d44ad65438 (table test-table [id=bb16a332a4e5488fbcb594d725b7664b]) successfully deleted
W20260620 22:01:35.225062 22473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:42537 (error 108)
I20260620 22:01:35.225219 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 22683
I20260620 22:01:35.225999 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.225894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42537 (local address 127.0.0.1:39480)
0620 22:01:35.225936 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:35.225969 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:01:35.226174 22473 catalog_manager.cc:4584] TS 250124f3d4184342b7dafbb0b48831c7 (127.6.241.129:42537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 082d73956d63480bac43b1d44ad65438: Network error: Client connection negotiation failed: client connection to 127.6.241.129:42537: connect: Connection refused (error 111)
I20260620 22:01:35.229640 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:35.229564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:34045 (local address 127.0.0.1:51878)
0620 22:01:35.229598 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:35.229621 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.130:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 22:01:35.229665 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 22817
I20260620 22:01:35.234140 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 22455
I20260620 22:01:55.239364 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41935
--webserver_interface=127.6.241.190
--webserver_port=43623
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:41935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:01:55.321460 22987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:55.321687 22987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:55.321725 22987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:55.322932 22987 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:01:55.322988 22987 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:01:55.323009 22987 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:01:55.323055 22987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:01:55.323103 22987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:55.323143 22987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:55.323156 22987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:01:55.323185 22987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:01:55.324689 22987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:41935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:55.324945 22987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:55.325193 22987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:55.333895 22994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:55.333914 22993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:55.333915 22987 server_base.cc:1034] running on GCE node
W20260620 22:01:55.333914 22996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:55.334460 22987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:55.334794 22987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:55.335945 22987 hybrid_clock.cc:648] HybridClock initialized: now 1781992915335936 us; error 29 us; skew 500 ppm
Jun 20 22:01:55 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992915, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:55.338506 22987 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:01:55.338996 22987 webserver.cc:460] Webserver started at http://127.6.241.190:43623/ using document root <none> and password file <none>
I20260620 22:01:55.339201 22987 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:55.339254 22987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:55.340633 22987 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:01:55.341269 23003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:55.341449 22987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:01:55.341557 22987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "e0d1cf518d8e4cd997670173b330c4a3"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "a1cca8f62efd328667757cc8c2134c7d"
server_key_iv: "e0427571b22e5617a60df2c55136f50d"
server_key_version: "encryptionkey@0"
I20260620 22:01:55.341967 22987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:55.362511 22987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:55.364954 22987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:55.365118 22987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:55.369295 22987 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:41935
I20260620 22:01:55.369328 23055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:41935 every 8 connection(s)
I20260620 22:01:55.369773 22987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:01:55.371364 23056 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:01:55.372049 23056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Bootstrap starting.
I20260620 22:01:55.374491 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 22987
I20260620 22:01:55.374845 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:42537
--local_ip_for_outbound_sockets=127.6.241.129
--tserver_master_addrs=127.6.241.190:41935
--webserver_port=41969
--webserver_interface=127.6.241.129
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:55.381175 23056 log.cc:824] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:55.383342 23056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:55.383687 23056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Bootstrap complete.
I20260620 22:01:55.386145 23056 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:01:55.386409 23056 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d1cf518d8e4cd997670173b330c4a3, State: Initialized, Role: FOLLOWER
I20260620 22:01:55.386636 23056 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:01:55.386720 23056 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:01:55.386778 23056 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:01:55.386830 23056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:01:55.387953 23056 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:01:55.388083 23056 leader_election.cc:302] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d1cf518d8e4cd997670173b330c4a3; no voters:
I20260620 22:01:55.388303 23056 leader_election.cc:288] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:01:55.388365 23060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:01:55.388545 23060 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 2 LEADER]: Becoming Leader. State: Replica: e0d1cf518d8e4cd997670173b330c4a3, State: Running, Role: LEADER
I20260620 22:01:55.388703 23056 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:01:55.388712 23060 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:01:55.390497 23062 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d1cf518d8e4cd997670173b330c4a3. Latest consensus state: current_term: 2 leader_uuid: "e0d1cf518d8e4cd997670173b330c4a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } } }
I20260620 22:01:55.390589 23062 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:55.390983 23061 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0d1cf518d8e4cd997670173b330c4a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } } }
I20260620 22:01:55.391013 23074 catalog_manager.cc:1259] Loaded cluster ID: 43fad989ceb5452aab30b17574988905
I20260620 22:01:55.391043 23061 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:01:55.391058 23074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: loading cluster ID for follower catalog manager: success
I20260620 22:01:55.392292 23074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: acquiring CA information for follower catalog manager: success
I20260620 22:01:55.392735 23074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:01:55.392819 23076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:01:55.393262 23076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:01:55.393373 23076 catalog_manager.cc:1259] Loaded cluster ID: 43fad989ceb5452aab30b17574988905
I20260620 22:01:55.393409 23076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:01:55.393587 23076 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:01:55.393677 23076 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Loaded TSK: 0
I20260620 22:01:55.402748 23076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Generated new TSK 1
I20260620 22:01:55.402865 23076 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:01:55.460050 23058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:55.460283 23058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:55.460323 23058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:55.461498 23058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:55.461563 23058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:55.461608 23058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:01:55.463083 23058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:42537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=41969
--tserver_master_addrs=127.6.241.190:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:55.463337 23058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:55.463575 23058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:55.468499 23084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:55.468524 23083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:55.468743 23086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:55.469185 23058 server_base.cc:1034] running on GCE node
I20260620 22:01:55.469374 23058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:55.469645 23058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:55.470799 23058 hybrid_clock.cc:648] HybridClock initialized: now 1781992915470784 us; error 33 us; skew 500 ppm
Jun 20 22:01:55 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992915, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:55.473531 23058 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:01:55.474007 23058 webserver.cc:460] Webserver started at http://127.6.241.129:41969/ using document root <none> and password file <none>
I20260620 22:01:55.474253 23058 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:55.474339 23058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:55.475903 23058 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:55.476673 23093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:55.476884 23058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:55.476987 23058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "250124f3d4184342b7dafbb0b48831c7"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "8dc390df67b0a5016475542461939eb3"
server_key_iv: "e2d744939b33930626c0e3fff1705e34"
server_key_version: "encryptionkey@0"
I20260620 22:01:55.477375 23058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:55.496821 23058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:55.499173 23058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:55.499342 23058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:55.499596 23058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:55.500156 23100 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:01:55.502087 23058 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:01:55.502151 23058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:01:55.502198 23058 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:01:55.502837 23058 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:01:55.502903 23058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:55.502960 23100 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Bootstrap starting.
I20260620 22:01:55.510234 23058 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:42537
I20260620 22:01:55.510278 23207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:42537 every 8 connection(s)
I20260620 22:01:55.510756 23058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:01:55.513754 23100 log.cc:824] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:55.516088 23100 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:55.516376 23100 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Bootstrap complete.
I20260620 22:01:55.517475 23100 ts_tablet_manager.cc:1397] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260620 22:01:55.518896 23100 raft_consensus.cc:348] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:55.519153 23100 raft_consensus.cc:729] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250124f3d4184342b7dafbb0b48831c7, State: Initialized, Role: FOLLOWER
I20260620 22:01:55.519310 23100 consensus_queue.cc:260] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:55.519604 23100 ts_tablet_manager.cc:1428] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:01:55.520329 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 23058
I20260620 22:01:55.520609 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:34045
--local_ip_for_outbound_sockets=127.6.241.130
--tserver_master_addrs=127.6.241.190:41935
--webserver_port=44471
--webserver_interface=127.6.241.130
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:01:55 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992915, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:55.529049 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.511582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:56133 (local address 127.6.241.190:41935)
0620 22:01:55.514204 (+ 2622us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.514213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.514256 (+ 43us) server_negotiation.cc:407] Connection header received
0620 22:01:55.514386 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.514403 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.514456 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.514620 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:55.515449 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.516865 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.518482 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.519180 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.519292 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.522021 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:55.522027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:55.522027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:55.522043 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:55.527349 (+ 5306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.527792 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.527795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.527795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.527899 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.528150 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.528151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.528152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.528366 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:55.528461 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.528739 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.528861 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2571,"thread_start_us":80,"threads_started":1}
I20260620 22:01:55.529062 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.511514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:56133)
0620 22:01:55.511792 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:55.511830 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:55.512256 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:55.514690 (+ 2434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:55.514694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:55.515053 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:55.515303 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.515309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.516992 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.516993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.518384 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.518387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.519400 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.519401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.519536 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.520068 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:55.520077 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:55.521881 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:55.527461 (+ 5580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:55.527468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:55.527470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:55.527695 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:55.527975 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:55.527976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:55.527977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:55.528081 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:55.528442 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:55.528448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:55.528579 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 22:01:55.528687 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 22:01:55.528853 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
I20260620 22:01:55.530128 23208 heartbeater.cc:347] Connected to a master server at 127.6.241.190:41935
I20260620 22:01:55.530225 23208 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:55.530396 23208 heartbeater.cc:511] Master 127.6.241.190:41935 requested a full tablet report, sending...
I20260620 22:01:55.531129 23020 ts_manager.cc:194] Registered new tserver with Master: 250124f3d4184342b7dafbb0b48831c7 (127.6.241.129:42537)
I20260620 22:01:55.532342 23020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:56133
I20260620 22:01:55.533208 23208 heartbeater.cc:503] Master 127.6.241.190:41935 was elected leader, sending a full tablet report...
I20260620 22:01:55.539162 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.531693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42537 (local address 127.0.0.1:46436)
0620 22:01:55.532031 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:55.532065 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:55.532121 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:55.535904 (+ 3783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:55.535907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:55.535909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:55.535958 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.535959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.537233 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.537233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.537986 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.537988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.538954 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.538955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.539011 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.539021 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 22:01:55.539076 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:01:55.539114 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":76,"threads_started":1}
I20260620 22:01:55.539234 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.531907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.6.241.129:42537)
0620 22:01:55.535722 (+ 3815us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.535726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.535737 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:01:55.535769 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.535771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.535786 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.535869 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:55.536041 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.537155 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.538082 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.538854 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.538943 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.538993 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.539116 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.539153 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3785,"thread_start_us":58,"threads_started":1}
I20260620 22:01:55.539615 23126 tablet_service.cc:1514] Processing DeleteTablet for tablet 082d73956d63480bac43b1d44ad65438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:35 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:46436
I20260620 22:01:55.540169 23217 tablet_replica.cc:330] stopping tablet replica
I20260620 22:01:55.540271 23217 raft_consensus.cc:2229] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:55.540334 23217 raft_consensus.cc:2258] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:55.540623 23217 ts_tablet_manager.cc:1905] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:55.543155 23217 ts_tablet_manager.cc:1918] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:55.543236 23217 log.cc:1196] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal/wals/082d73956d63480bac43b1d44ad65438
I20260620 22:01:55.543464 23217 ts_tablet_manager.cc:1939] T 082d73956d63480bac43b1d44ad65438 P 250124f3d4184342b7dafbb0b48831c7: Deleting consensus metadata
I20260620 22:01:55.544029 23006 catalog_manager.cc:4841] TS 250124f3d4184342b7dafbb0b48831c7 (127.6.241.129:42537): tablet 082d73956d63480bac43b1d44ad65438 (table test-table [id=bb16a332a4e5488fbcb594d725b7664b]) successfully deleted
W20260620 22:01:55.606395 23214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:55.606611 23214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:55.606667 23214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:55.607801 23214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:55.607841 23214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:55.607867 23214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:01:55.609212 23214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:34045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=44471
--tserver_master_addrs=127.6.241.190:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:55.609386 23214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:55.609555 23214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:55.614298 23223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:55.614298 23225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:55.614313 23214 server_base.cc:1034] running on GCE node
W20260620 22:01:55.614308 23222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:55.614822 23214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:55.615020 23214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:55.616180 23214 hybrid_clock.cc:648] HybridClock initialized: now 1781992915616181 us; error 36 us; skew 500 ppm
Jun 20 22:01:55 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992915, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:55.618501 23214 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:01:55.618937 23214 webserver.cc:460] Webserver started at http://127.6.241.130:44471/ using document root <none> and password file <none>
I20260620 22:01:55.619109 23214 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:55.619163 23214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:55.620416 23214 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:01:55.621006 23232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:55.621181 23214 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:01:55.621263 23214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "63c905d15cd946ba901971361636cec9"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "f2b7d4c48784f4821aa304a8ef0880a4"
server_key_iv: "082db7d9d10695576fe957a72501c3a9"
server_key_version: "encryptionkey@0"
I20260620 22:01:55.621615 23214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:55.634569 23214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:55.636690 23214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:55.636875 23214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:55.637141 23214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:55.637579 23239 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:01:55.638921 23214 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:01:55.638979 23214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260620 22:01:55.639019 23214 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:01:55.639611 23214 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:01:55.639667 23214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:01:55.639699 23239 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Bootstrap starting.
I20260620 22:01:55.646195 23214 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:34045
I20260620 22:01:55.646270 23346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:34045 every 8 connection(s)
I20260620 22:01:55.646552 23214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:01:55.650305 23239 log.cc:824] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:55.652837 23239 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:01:55.653136 23239 tablet_bootstrap.cc:492] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Bootstrap complete.
Jun 20 22:01:55 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992915, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:55.654068 23239 ts_tablet_manager.cc:1397] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 22:01:55.655001 23239 raft_consensus.cc:348] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:55.655244 23239 raft_consensus.cc:729] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c905d15cd946ba901971361636cec9, State: Initialized, Role: FOLLOWER
I20260620 22:01:55.655436 23239 consensus_queue.cc:260] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } }
I20260620 22:01:55.655721 23239 ts_tablet_manager.cc:1428] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260620 22:01:55.655974 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 23214
I20260620 22:01:55.656235 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:36041
--local_ip_for_outbound_sockets=127.6.241.131
--tserver_master_addrs=127.6.241.190:41935
--webserver_port=36681
--webserver_interface=127.6.241.131
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:01:55.656739 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.647226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:34765 (local address 127.6.241.190:41935)
0620 22:01:55.647273 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.647275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.647552 (+ 277us) server_negotiation.cc:407] Connection header received
0620 22:01:55.648325 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.648326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.648339 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.648377 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:55.649297 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.649778 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.651261 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.651459 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.651514 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.653804 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:55.653805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:55.653806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:55.653813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:55.655210 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.655636 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.655637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.655637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.655700 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.656007 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.656008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.656009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.656113 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:55.656169 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.656486 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.656604 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:01:55.656903 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.647209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:34765)
0620 22:01:55.647405 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:55.647454 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:55.648223 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:55.648457 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:55.648462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:55.648911 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:55.649211 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.649218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.649893 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.649894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.651174 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.651177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.651562 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.651563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.651692 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.652173 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:55.652183 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:55.653712 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:55.655307 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:55.655314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:55.655315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:55.655574 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:55.655792 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:55.655793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:55.655794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:55.655933 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:55.656182 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:55.656187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:55.656344 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 22:01:55.656464 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 22:01:55.656647 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260620 22:01:55.658056 23347 heartbeater.cc:347] Connected to a master server at 127.6.241.190:41935
I20260620 22:01:55.658159 23347 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:55.658327 23347 heartbeater.cc:511] Master 127.6.241.190:41935 requested a full tablet report, sending...
I20260620 22:01:55.658840 23020 ts_manager.cc:194] Registered new tserver with Master: 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045)
I20260620 22:01:55.659410 23020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:34765
I20260620 22:01:55.660175 23347 heartbeater.cc:503] Master 127.6.241.190:41935 was elected leader, sending a full tablet report...
Jun 20 22:01:55 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992915, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.130@KRBTEST.COM
I20260620 22:01:55.667265 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.659110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:34045 (local address 127.0.0.1:49552)
0620 22:01:55.659212 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:55.659221 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:55.659255 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:55.659574 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:55.659575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:55.659909 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:55.659945 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.659947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.661278 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.661279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.661747 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.661748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.662397 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.662398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.662446 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.662923 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:55.662928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:55.664143 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:55.666202 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:55.666208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:55.666210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:55.666389 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:55.666651 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:55.666652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:55.666652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:55.666705 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:55.667035 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:55.667040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:55.667080 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 22:01:55.667132 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:01:55.667203 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 22:01:55.667395 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.659231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.6.241.130:34045)
0620 22:01:55.659404 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.659409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.659420 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:01:55.659452 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.659454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.659467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.659568 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:55.660030 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.661197 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.661825 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.662310 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.662402 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.664227 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:55.664235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:55.664235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:55.664243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:55.666122 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.666471 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.666475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.666476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.666540 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.666770 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.666772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.666772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.666961 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:55.667057 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.667194 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.667295 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":68,"threads_started":1}
I20260620 22:01:55.667711 23281 tablet_service.cc:1514] Processing DeleteTablet for tablet 082d73956d63480bac43b1d44ad65438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:35 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:49552
I20260620 22:01:55.668222 23354 tablet_replica.cc:330] stopping tablet replica
I20260620 22:01:55.668321 23354 raft_consensus.cc:2229] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:55.668406 23354 raft_consensus.cc:2258] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:55.668679 23354 ts_tablet_manager.cc:1905] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:55.670562 23354 ts_tablet_manager.cc:1918] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:55.670667 23354 log.cc:1196] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/082d73956d63480bac43b1d44ad65438
I20260620 22:01:55.670886 23354 ts_tablet_manager.cc:1939] T 082d73956d63480bac43b1d44ad65438 P 63c905d15cd946ba901971361636cec9: Deleting consensus metadata
I20260620 22:01:55.671316 23005 catalog_manager.cc:4841] TS 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045): tablet 082d73956d63480bac43b1d44ad65438 (table test-table [id=bb16a332a4e5488fbcb594d725b7664b]) successfully deleted
W20260620 22:01:55.739323 23352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:01:55.739574 23352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:01:55.739614 23352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:01:55.740752 23352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:01:55.740815 23352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:01:55.740864 23352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:01:55.742261 23352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:36041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=36681
--tserver_master_addrs=127.6.241.190:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:01:55.742522 23352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:01:55.742769 23352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:01:55.747861 23362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:55.747819 23360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:01:55.747813 23359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:01:55.747905 23352 server_base.cc:1034] running on GCE node
I20260620 22:01:55.748202 23352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:01:55.748453 23352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:01:55.749588 23352 hybrid_clock.cc:648] HybridClock initialized: now 1781992915749571 us; error 34 us; skew 500 ppm
Jun 20 22:01:55 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992915, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:01:55.752022 23352 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:01:55.752427 23352 webserver.cc:460] Webserver started at http://127.6.241.131:36681/ using document root <none> and password file <none>
I20260620 22:01:55.752636 23352 fs_manager.cc:359] Metadata directory not provided
I20260620 22:01:55.752707 23352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:01:55.753973 23352 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:01:55.754528 23369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:55.754698 23352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:01:55.754792 23352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "eeb4c1a2b64847d284ab66ffd2c457d1"
format_stamp: "Formatted at 2026-06-20 22:01:35 on dist-test-slave-gl2g"
server_key: "35d5e6eecf49cb09e82fdf88d7a9a9b5"
server_key_iv: "71da8ee5d1097ea9e1dfeaa332c13595"
server_key_version: "encryptionkey@0"
I20260620 22:01:55.755105 23352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:01:55.789388 23352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:01:55.791517 23352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:01:55.791749 23352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:01:55.791996 23352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:01:55.792301 23352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:01:55.792353 23352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:55.792395 23352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:01:55.792429 23352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:55.797856 23352 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:36041
I20260620 22:01:55.797960 23482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:36041 every 8 connection(s)
I20260620 22:01:55.798270 23352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:01:55.801710 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 23352
Jun 20 22:01:55 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992915, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:01:55.807869 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.798991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:49155 (local address 127.6.241.190:41935)
0620 22:01:55.799049 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.799051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.799256 (+ 205us) server_negotiation.cc:407] Connection header received
0620 22:01:55.799743 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.799744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.799753 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.799804 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:55.800687 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.801167 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.802286 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.802636 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.802699 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.804938 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:55.804940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:55.804941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:55.804948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:55.806305 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.806780 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.806781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.806781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.806843 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.807165 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.807165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.807166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.807266 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:55.807356 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.807624 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.807778 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":31}
I20260620 22:01:55.808017 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.798994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:49155)
0620 22:01:55.799142 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:55.799182 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:55.799663 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:55.799868 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:55.799871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:55.800432 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:01:55.800615 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.800619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.801243 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.801243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.802186 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.802190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.802716 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.802717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.802878 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.803441 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:01:55.803449 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:01:55.804824 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:01:55.806409 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:55.806415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:55.806417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:55.806696 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:55.806912 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:01:55.806913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:01:55.806913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:01:55.807026 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:01:55.807346 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:01:55.807353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:01:55.807494 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 22:01:55.807601 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 22:01:55.807776 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260620 22:01:55.808521 23483 heartbeater.cc:347] Connected to a master server at 127.6.241.190:41935
I20260620 22:01:55.808641 23483 heartbeater.cc:464] Registering TS with master...
I20260620 22:01:55.808844 23483 heartbeater.cc:511] Master 127.6.241.190:41935 requested a full tablet report, sending...
I20260620 22:01:55.809128 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.802131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.6.241.190:41935)
0620 22:01:55.802257 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.802259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.802320 (+ 61us) server_negotiation.cc:407] Connection header received
0620 22:01:55.802378 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.802378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.802389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.802451 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:01:55.803199 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.803729 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.804596 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.804805 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.804891 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.806323 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:01:55.806324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:01:55.806324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:01:55.806331 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:01:55.807788 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.808342 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.808343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.808343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.808415 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:01:55.808604 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:01:55.808605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:01:55.808605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:01:55.808735 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:01:55.808904 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.808945 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.809024 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260620 22:01:55.809235 23020 ts_manager.cc:194] Registered new tserver with Master: eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041)
I20260620 22:01:55.809633 23020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:49155
I20260620 22:01:55.809813 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:01:55.812309 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.810159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.6.241.190:41935)
0620 22:01:55.810199 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.810201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.810267 (+ 66us) server_negotiation.cc:407] Connection header received
0620 22:01:55.810359 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.810360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.810397 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.810467 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:01:55.810564 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.811056 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.811737 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.811917 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.811966 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.812084 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:01:55.812166 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:01:55.812215 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.812249 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.812264 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:01:55.812572 23020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:01:55.814162 23020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:01:55.816056 23281 tablet_service.cc:1467] Processing CreateTablet for tablet d64f3550d9be4874a6d2af743731d24e (DEFAULT_TABLE table=test-table [id=557b1515c36745f28af6561b03f5c5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:55.816082 23126 tablet_service.cc:1467] Processing CreateTablet for tablet d64f3550d9be4874a6d2af743731d24e (DEFAULT_TABLE table=test-table [id=557b1515c36745f28af6561b03f5c5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:55.816241 23281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64f3550d9be4874a6d2af743731d24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:55.816260 23126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64f3550d9be4874a6d2af743731d24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:55.817584 23239 tablet_bootstrap.cc:492] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: Bootstrap starting.
I20260620 22:01:55.817653 23100 tablet_bootstrap.cc:492] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Bootstrap starting.
I20260620 22:01:55.817930 23239 tablet_bootstrap.cc:654] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:55.818082 23100 tablet_bootstrap.cc:654] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:55.818790 23100 tablet_bootstrap.cc:492] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: No bootstrap required, opened a new log
I20260620 22:01:55.818861 23100 ts_tablet_manager.cc:1397] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:01:55.818998 23100 raft_consensus.cc:348] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:01:55.819063 23100 raft_consensus.cc:374] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:55.819099 23100 raft_consensus.cc:729] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250124f3d4184342b7dafbb0b48831c7, State: Initialized, Role: FOLLOWER
I20260620 22:01:55.819149 23100 consensus_queue.cc:260] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:01:55.819231 23100 ts_tablet_manager.cc:1428] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:01:55.819732 23239 tablet_bootstrap.cc:492] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: No bootstrap required, opened a new log
I20260620 22:01:55.819784 23239 ts_tablet_manager.cc:1397] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260620 22:01:55.820122 23239 raft_consensus.cc:348] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:01:55.820191 23239 raft_consensus.cc:374] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:55.820227 23239 raft_consensus.cc:729] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c905d15cd946ba901971361636cec9, State: Initialized, Role: FOLLOWER
I20260620 22:01:55.820277 23239 consensus_queue.cc:260] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:01:55.820363 23239 ts_tablet_manager.cc:1428] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:01:55.821530 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.815853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:36041 (local address 127.0.0.1:49412)
0620 22:01:55.815886 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:55.815897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:55.815932 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:55.818047 (+ 2115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:55.818048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:55.818052 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:55.818095 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.818097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.819346 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.819346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.820418 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.820421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.821356 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.821357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.821406 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.821408 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:01:55.821455 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:01:55.821486 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 22:01:55.821549 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.817630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.6.241.131:36041)
0620 22:01:55.817860 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.817865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.817875 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:01:55.817913 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.817915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.817943 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.818028 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:55.818173 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.819262 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.820564 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.821287 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.821341 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.821379 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.821466 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.821491 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260620 22:01:55.822229 23417 tablet_service.cc:1467] Processing CreateTablet for tablet d64f3550d9be4874a6d2af743731d24e (DEFAULT_TABLE table=test-table [id=557b1515c36745f28af6561b03f5c5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:01:55.822496 23417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64f3550d9be4874a6d2af743731d24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:01:55.825254 23492 tablet_bootstrap.cc:492] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: Bootstrap starting.
I20260620 22:01:55.825878 23492 tablet_bootstrap.cc:654] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: Neither blocks nor log segments found. Creating new log.
I20260620 22:01:55.826327 23492 log.cc:824] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:01:55.827015 23492 tablet_bootstrap.cc:492] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: No bootstrap required, opened a new log
I20260620 22:01:55.827071 23492 ts_tablet_manager.cc:1397] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:01:55.828469 23492 raft_consensus.cc:348] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:01:55.828603 23492 raft_consensus.cc:374] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:01:55.828625 23492 raft_consensus.cc:729] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb4c1a2b64847d284ab66ffd2c457d1, State: Initialized, Role: FOLLOWER
I20260620 22:01:55.828733 23492 consensus_queue.cc:260] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:01:55.828960 23492 ts_tablet_manager.cc:1428] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:01:55.829025 23483 heartbeater.cc:503] Master 127.6.241.190:41935 was elected leader, sending a full tablet report...
W20260620 22:01:55.897150 23348 tablet.cc:2369] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:01:55.928633 23213 raft_consensus.cc:482] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:01:55.928759 23213 raft_consensus.cc:504] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:01:55.929100 23213 leader_election.cc:288] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041), 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045)
I20260620 22:01:55.931950 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.929366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:36421 (local address 127.6.241.131:36041)
0620 22:01:55.929417 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.929419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.929429 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:01:55.929459 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.929460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.929469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.929555 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:55.929720 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.930196 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.931224 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.931598 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.931691 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.931700 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.931847 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.931887 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:01:55.931972 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.929279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:36041 (local address 127.6.241.129:36421)
0620 22:01:55.929329 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:55.929341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:55.929389 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:55.929559 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:55.929560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:55.929562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:55.929644 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.929646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.930279 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.930280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.931120 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.931123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.931690 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.931691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.931771 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.931774 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:01:55.931839 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 22:01:55.931880 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 22:01:55.932287 23437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64f3550d9be4874a6d2af743731d24e" candidate_uuid: "250124f3d4184342b7dafbb0b48831c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" is_pre_election: true
I20260620 22:01:55.932461 23437 raft_consensus.cc:2454] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 250124f3d4184342b7dafbb0b48831c7 in term 0.
I20260620 22:01:55.932478 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.929409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:58255 (local address 127.6.241.130:34045)
0620 22:01:55.929451 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.929453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.929571 (+ 118us) server_negotiation.cc:407] Connection header received
0620 22:01:55.929631 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.929632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.929657 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.929736 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:01:55.930150 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.930763 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.931644 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.932089 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.932164 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.932237 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.932365 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.932403 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:01:55.932477 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.929298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:34045 (local address 127.6.241.129:58255)
0620 22:01:55.929509 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:55.929516 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:01:55.929577 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:01:55.929740 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:01:55.929741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:01:55.929742 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:01:55.930073 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.930074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.930844 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.930844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.931555 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:01:55.931557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.932201 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.932202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:01:55.932274 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.932283 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 22:01:55.932347 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 22:01:55.932384 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":54,"thread_start_us":106,"threads_started":1}
I20260620 22:01:55.932677 23097 leader_election.cc:302] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250124f3d4184342b7dafbb0b48831c7, eeb4c1a2b64847d284ab66ffd2c457d1; no voters:
I20260620 22:01:55.932765 23301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64f3550d9be4874a6d2af743731d24e" candidate_uuid: "250124f3d4184342b7dafbb0b48831c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c905d15cd946ba901971361636cec9" is_pre_election: true
I20260620 22:01:55.932821 23213 raft_consensus.cc:2781] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:01:55.932878 23213 raft_consensus.cc:482] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:01:55.932909 23213 raft_consensus.cc:3037] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:55.932909 23301 raft_consensus.cc:2454] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 250124f3d4184342b7dafbb0b48831c7 in term 0.
I20260620 22:01:55.933634 23213 raft_consensus.cc:504] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:01:55.933781 23213 leader_election.cc:288] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 1 election: Requested vote from peers eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041), 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045)
I20260620 22:01:55.933923 23301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64f3550d9be4874a6d2af743731d24e" candidate_uuid: "250124f3d4184342b7dafbb0b48831c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c905d15cd946ba901971361636cec9"
I20260620 22:01:55.933941 23437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64f3550d9be4874a6d2af743731d24e" candidate_uuid: "250124f3d4184342b7dafbb0b48831c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1"
I20260620 22:01:55.934017 23437 raft_consensus.cc:3037] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:55.934020 23301 raft_consensus.cc:3037] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:01:55.934888 23437 raft_consensus.cc:2454] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250124f3d4184342b7dafbb0b48831c7 in term 1.
I20260620 22:01:55.934983 23301 raft_consensus.cc:2454] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250124f3d4184342b7dafbb0b48831c7 in term 1.
I20260620 22:01:55.935055 23097 leader_election.cc:302] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250124f3d4184342b7dafbb0b48831c7, eeb4c1a2b64847d284ab66ffd2c457d1; no voters:
I20260620 22:01:55.935189 23213 raft_consensus.cc:2781] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:01:55.935355 23213 raft_consensus.cc:686] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 LEADER]: Becoming Leader. State: Replica: 250124f3d4184342b7dafbb0b48831c7, State: Running, Role: LEADER
I20260620 22:01:55.935439 23213 consensus_queue.cc:237] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:01:55.936164 23020 catalog_manager.cc:5495] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 250124f3d4184342b7dafbb0b48831c7 (127.6.241.129). New cstate: current_term: 1 leader_uuid: "250124f3d4184342b7dafbb0b48831c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } health_report { overall_health: HEALTHY } } }
I20260620 22:01:55.967890 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.965496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.6.241.129:42537)
0620 22:01:55.965562 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:01:55.965564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:01:55.965582 (+ 18us) server_negotiation.cc:407] Connection header received
0620 22:01:55.965618 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:01:55.965618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:01:55.965645 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:01:55.965720 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:01:55.965867 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.966345 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.967105 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:01:55.967316 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:01:55.967392 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:01:55.967559 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:01:55.967656 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:01:55.967741 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:01:55.967780 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 22:01:55.967802 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:01:55.969398 23301 raft_consensus.cc:1261] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Refusing update from remote peer 250124f3d4184342b7dafbb0b48831c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:55.969398 23437 raft_consensus.cc:1261] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [term 1 FOLLOWER]: Refusing update from remote peer 250124f3d4184342b7dafbb0b48831c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:01:55.969702 23213 consensus_queue.cc:1035] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:55.969841 23496 consensus_queue.cc:1035] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:01:55.972699 23501 mvcc.cc:205] Tried to move back new op lower bound from 7299042983807959040 to 7299042983672123392. Current Snapshot: MvccSnapshot[applied={T|T < 7299042983807959040}]
I20260620 22:01:55.973810 23020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:55.973897 23020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:01:55.974583 23020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Sending DeleteTablet for 3 replicas of tablet d64f3550d9be4874a6d2af743731d24e
I20260620 22:01:55.974895 23126 tablet_service.cc:1514] Processing DeleteTablet for tablet d64f3550d9be4874a6d2af743731d24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:55 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:46436
I20260620 22:01:55.974927 23417 tablet_service.cc:1514] Processing DeleteTablet for tablet d64f3550d9be4874a6d2af743731d24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:55 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:49412
I20260620 22:01:55.974987 23217 tablet_replica.cc:330] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: stopping tablet replica
I20260620 22:01:55.975041 23217 raft_consensus.cc:2229] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:01:55.975083 23281 tablet_service.cc:1514] Processing DeleteTablet for tablet d64f3550d9be4874a6d2af743731d24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:01:55 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:49552
I20260620 22:01:55.975123 23217 raft_consensus.cc:2258] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:55.975167 23354 tablet_replica.cc:330] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: stopping tablet replica
I20260620 22:01:55.975217 23354 raft_consensus.cc:2229] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:55.975284 23354 raft_consensus.cc:2258] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:55.975462 23217 ts_tablet_manager.cc:1905] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:55.975524 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 23058
I20260620 22:01:55.975659 23354 ts_tablet_manager.cc:1905] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:55.976109 23509 tablet_replica.cc:330] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: stopping tablet replica
I20260620 22:01:55.976195 23509 raft_consensus.cc:2229] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:01:55.976284 23509 raft_consensus.cc:2258] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:01:55.977401 23509 ts_tablet_manager.cc:1905] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:01:55.977785 23354 ts_tablet_manager.cc:1918] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:55.977847 23354 log.cc:1196] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/d64f3550d9be4874a6d2af743731d24e
I20260620 22:01:55.978084 23354 ts_tablet_manager.cc:1939] T d64f3550d9be4874a6d2af743731d24e P 63c905d15cd946ba901971361636cec9: Deleting consensus metadata
I20260620 22:01:55.978411 23005 catalog_manager.cc:4841] TS 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045): tablet d64f3550d9be4874a6d2af743731d24e (table test-table [id=557b1515c36745f28af6561b03f5c5cf]) successfully deleted
I20260620 22:01:55.978837 23509 ts_tablet_manager.cc:1918] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:01:55.978893 23509 log.cc:1196] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/d64f3550d9be4874a6d2af743731d24e
I20260620 22:01:55.979153 23509 ts_tablet_manager.cc:1939] T d64f3550d9be4874a6d2af743731d24e P eeb4c1a2b64847d284ab66ffd2c457d1: Deleting consensus metadata
I20260620 22:01:55.979527 23007 catalog_manager.cc:4841] TS eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041): tablet d64f3550d9be4874a6d2af743731d24e (table test-table [id=557b1515c36745f28af6561b03f5c5cf]) successfully deleted
W20260620 22:01:55.981179 23006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:42537 (error 108)
I20260620 22:01:55.981664 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:01:55.981560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42537 (local address 127.0.0.1:46458)
0620 22:01:55.981607 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:01:55.981640 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:01:55.981768 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 23214
W20260620 22:01:55.981786 23006 catalog_manager.cc:4584] TS 250124f3d4184342b7dafbb0b48831c7 (127.6.241.129:42537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d64f3550d9be4874a6d2af743731d24e: Network error: Client connection negotiation failed: client connection to 127.6.241.129:42537: connect: Connection refused (error 111)
I20260620 22:01:55.986801 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 23352
I20260620 22:01:55.991659 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 22987
I20260620 22:02:15.997058 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41935
--webserver_interface=127.6.241.190
--webserver_port=43623
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:41935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:16.074815 23511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:16.075019 23511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:16.075042 23511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:16.076223 23511 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:02:16.076265 23511 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:02:16.076278 23511 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:02:16.076330 23511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:16.076357 23511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:16.076381 23511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:16.076411 23511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:16.076437 23511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:16.077904 23511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:41935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:16.078161 23511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:16.078383 23511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:16.083442 23516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:16.083492 23511 server_base.cc:1034] running on GCE node
W20260620 22:02:16.083458 23519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:16.083441 23517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:16.083956 23511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:16.084194 23511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:16.085335 23511 hybrid_clock.cc:648] HybridClock initialized: now 1781992936085321 us; error 25 us; skew 500 ppm
Jun 20 22:02:16 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:16.087795 23511 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:02:16.088153 23511 webserver.cc:460] Webserver started at http://127.6.241.190:43623/ using document root <none> and password file <none>
I20260620 22:02:16.088341 23511 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:16.088382 23511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:16.089596 23511 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:16.090194 23526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:16.090397 23511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:16.090498 23511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "e0d1cf518d8e4cd997670173b330c4a3"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "a1cca8f62efd328667757cc8c2134c7d"
server_key_iv: "e0427571b22e5617a60df2c55136f50d"
server_key_version: "encryptionkey@0"
I20260620 22:02:16.090876 23511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:16.097064 23511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:16.098728 23511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:16.098868 23511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:16.102409 23511 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:41935
I20260620 22:02:16.102460 23578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:41935 every 8 connection(s)
I20260620 22:02:16.102838 23511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:16.104192 23579 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:02:16.104821 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Bootstrap starting.
I20260620 22:02:16.111450 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 23511
I20260620 22:02:16.111809 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:42537
--local_ip_for_outbound_sockets=127.6.241.129
--tserver_master_addrs=127.6.241.190:41935
--webserver_port=41969
--webserver_interface=127.6.241.129
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:02:16.113457 23579 log.cc:824] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:16.115643 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:02:16.115928 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Bootstrap complete.
I20260620 22:02:16.118067 23579 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:02:16.118280 23579 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d1cf518d8e4cd997670173b330c4a3, State: Initialized, Role: FOLLOWER
I20260620 22:02:16.118430 23579 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:02:16.118489 23579 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:16.118582 23579 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:16.118647 23579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 2 FOLLOWER]: Advancing to term 3
I20260620 22:02:16.119616 23579 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:02:16.119699 23579 leader_election.cc:302] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d1cf518d8e4cd997670173b330c4a3; no voters:
I20260620 22:02:16.119904 23579 leader_election.cc:288] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 22:02:16.119936 23583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 22:02:16.120047 23583 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [term 3 LEADER]: Becoming Leader. State: Replica: e0d1cf518d8e4cd997670173b330c4a3, State: Running, Role: LEADER
I20260620 22:02:16.120128 23583 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } }
I20260620 22:02:16.120157 23579 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:16.120424 23585 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d1cf518d8e4cd997670173b330c4a3. Latest consensus state: current_term: 3 leader_uuid: "e0d1cf518d8e4cd997670173b330c4a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } } }
I20260620 22:02:16.120523 23585 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:16.121044 23587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:16.121778 23587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:16.121829 23584 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e0d1cf518d8e4cd997670173b330c4a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d1cf518d8e4cd997670173b330c4a3" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41935 } } }
I20260620 22:02:16.121887 23587 catalog_manager.cc:1259] Loaded cluster ID: 43fad989ceb5452aab30b17574988905
I20260620 22:02:16.121903 23584 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:16.121914 23587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:16.123067 23587 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:16.123235 23587 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Loaded TSK: 0
I20260620 22:02:16.123282 23587 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0d1cf518d8e4cd997670173b330c4a3: Loaded TSK: 1
F20260620 22:02:16.123519 23587 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f73a63859ad google::LogMessage::Fail() at ??:0
@ 0x7f73a638888a google::LogMessage::SendToLog() at ??:0
@ 0x7f73a63854de google::LogMessage::Flush() at ??:0
@ 0x7f73a6386a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f73a8af2104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f73a8aefd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f73ae2e31cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f73ae2e348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f73ae2e3594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f73ae2a9c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f73ae2b3ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f73a82530d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f73a8248954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73a90ac6db start_thread at ??:0
@ 0x7f73a4e8361f clone at ??:0
W20260620 22:02:16.212154 23581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:16.212332 23581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:16.212353 23581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:16.213425 23581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:16.213464 23581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:16.213490 23581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:02:16.214958 23581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:42537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=41969
--tserver_master_addrs=127.6.241.190:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:16.215181 23581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:16.215396 23581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:16.220292 23621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:16.220330 23620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:16.220384 23581 server_base.cc:1034] running on GCE node
W20260620 22:02:16.220459 23623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:16.220675 23581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:16.220928 23581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:16.222077 23581 hybrid_clock.cc:648] HybridClock initialized: now 1781992936222071 us; error 30 us; skew 500 ppm
Jun 20 22:02:16 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:16.224323 23581 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:02:16.224771 23581 webserver.cc:460] Webserver started at http://127.6.241.129:41969/ using document root <none> and password file <none>
I20260620 22:02:16.224972 23581 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:16.225013 23581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:16.226249 23581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:02:16.227002 23630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:16.227147 23581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:16.227240 23581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "250124f3d4184342b7dafbb0b48831c7"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "8dc390df67b0a5016475542461939eb3"
server_key_iv: "e2d744939b33930626c0e3fff1705e34"
server_key_version: "encryptionkey@0"
I20260620 22:02:16.227528 23581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:16.244747 23581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:16.246941 23581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:16.247077 23581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:16.247300 23581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:16.247786 23637 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:02:16.249152 23581 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:02:16.249202 23581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:02:16.249225 23581 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:02:16.249815 23581 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:02:16.249857 23581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:02:16.249897 23637 tablet_bootstrap.cc:492] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Bootstrap starting.
I20260620 22:02:16.251796 23637 log.cc:824] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:16.255153 23637 tablet_bootstrap.cc:492] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:02:16.255476 23637 tablet_bootstrap.cc:492] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Bootstrap complete.
I20260620 22:02:16.255638 23637 ts_tablet_manager.cc:1397] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260620 22:02:16.257015 23581 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:42537
I20260620 22:02:16.257375 23581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:02:16.257761 23745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:42537 every 8 connection(s)
I20260620 22:02:16.257819 23637 raft_consensus.cc:348] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:02:16.258294 23637 raft_consensus.cc:729] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250124f3d4184342b7dafbb0b48831c7, State: Initialized, Role: FOLLOWER
I20260620 22:02:16.258427 23637 consensus_queue.cc:260] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:02:16.258701 23637 ts_tablet_manager.cc:1428] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260620 22:02:16.267115 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 23581
I20260620 22:02:16.267402 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:34045
--local_ip_for_outbound_sockets=127.6.241.130
--tserver_master_addrs=127.6.241.190:41935
--webserver_port=44471
--webserver_interface=127.6.241.130
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:16.306833 23748 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:02:16.257966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:39941)
0620 22:02:16.258174 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:16.258212 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:16.258844 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:16.306577 (+ 47733us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":67,"threads_started":1}
W20260620 22:02:16.307125 23746 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:41935 (0 consecutive failures): Network error: Failed to ping master at 127.6.241.190:41935: Client connection negotiation failed: client connection to 127.6.241.190:41935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:02:16.307464 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:16.307349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:55819)
0620 22:02:16.307400 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:16.307438 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 22:02:16.307789 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:16.307709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:44415)
0620 22:02:16.307750 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:16.307773 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:02:16.307919 23746 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:02:16.349309 23750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:16.349534 23750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:16.349573 23750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:16.350792 23750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:16.350847 23750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:16.350893 23750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:02:16.352381 23750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:34045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=44471
--tserver_master_addrs=127.6.241.190:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:16.352654 23750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:16.352881 23750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:16.357731 23756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:16.357758 23758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:16.357787 23750 server_base.cc:1034] running on GCE node
W20260620 22:02:16.357758 23755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:16.358139 23750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:16.358368 23750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:16.359508 23750 hybrid_clock.cc:648] HybridClock initialized: now 1781992936359483 us; error 44 us; skew 500 ppm
Jun 20 22:02:16 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:16.361857 23750 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:02:16.362285 23750 webserver.cc:460] Webserver started at http://127.6.241.130:44471/ using document root <none> and password file <none>
I20260620 22:02:16.362504 23750 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:16.362577 23750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:16.363919 23750 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:02:16.364434 23765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:16.364591 23750 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:02:16.364676 23750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "63c905d15cd946ba901971361636cec9"
format_stamp: "Formatted at 2026-06-20 22:01:34 on dist-test-slave-gl2g"
server_key: "f2b7d4c48784f4821aa304a8ef0880a4"
server_key_iv: "082db7d9d10695576fe957a72501c3a9"
server_key_version: "encryptionkey@0"
I20260620 22:02:16.364996 23750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:16.376021 23750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:16.378269 23750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:16.378417 23750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:16.378674 23750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:16.378990 23750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:16.379029 23750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:16.379050 23750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:16.379091 23750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:16.384949 23750 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:34045
I20260620 22:02:16.385017 23878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:34045 every 8 connection(s)
I20260620 22:02:16.385351 23750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:02:16.386317 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:16.385972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:43283)
0620 22:02:16.386114 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:16.386183 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
W20260620 22:02:16.386765 23879 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:41935 (0 consecutive failures): Network error: Failed to ping master at 127.6.241.190:41935: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
I20260620 22:02:16.387027 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:16.386937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:41745)
0620 22:02:16.386982 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:16.387006 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:02:16.387291 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:16.387210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:39827)
0620 22:02:16.387249 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:16.387270 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 22:02:16.387389 23879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:02:16.392280 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 23750
I20260620 22:02:16.392513 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:36041
--local_ip_for_outbound_sockets=127.6.241.131
--tserver_master_addrs=127.6.241.190:41935
--webserver_port=36681
--webserver_interface=127.6.241.131
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:16.473325 23882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:16.473520 23882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:16.473543 23882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:16.474721 23882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:16.474812 23882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:16.474851 23882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:02:16.476356 23882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:36041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=36681
--tserver_master_addrs=127.6.241.190:41935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:16.476614 23882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:16.476833 23882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:16.481549 23890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:16.481638 23888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:16.481549 23887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:16.481820 23882 server_base.cc:1034] running on GCE node
I20260620 22:02:16.482002 23882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:16.482213 23882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:16.483371 23882 hybrid_clock.cc:648] HybridClock initialized: now 1781992936483353 us; error 29 us; skew 500 ppm
Jun 20 22:02:16 dist-test-slave-gl2g krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:16.485905 23882 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:02:16.486327 23882 webserver.cc:460] Webserver started at http://127.6.241.131:36681/ using document root <none> and password file <none>
I20260620 22:02:16.486567 23882 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:16.486657 23882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:16.487948 23882 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:02:16.488524 23897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:16.488691 23882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:02:16.488793 23882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "eeb4c1a2b64847d284ab66ffd2c457d1"
format_stamp: "Formatted at 2026-06-20 22:01:35 on dist-test-slave-gl2g"
server_key: "35d5e6eecf49cb09e82fdf88d7a9a9b5"
server_key_iv: "71da8ee5d1097ea9e1dfeaa332c13595"
server_key_version: "encryptionkey@0"
I20260620 22:02:16.489121 23882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:16.503355 23882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:16.505730 23882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:16.505894 23882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:16.506124 23882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:16.506430 23882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:16.506487 23882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:16.506524 23882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:16.506556 23882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:16.512331 23882 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:36041
I20260620 22:02:16.512423 24010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:36041 every 8 connection(s)
I20260620 22:02:16.512738 23882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:02:16.513787 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:16.513415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:58693)
0620 22:02:16.513574 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:16.513642 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
W20260620 22:02:16.514196 24011 heartbeater.cc:644] Failed to heartbeat to 127.6.241.190:41935 (0 consecutive failures): Network error: Failed to ping master at 127.6.241.190:41935: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
I20260620 22:02:16.514460 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:16.514374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:36079)
0620 22:02:16.514415 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:16.514440 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:02:16.514750 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:16.514672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:60467)
0620 22:02:16.514711 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:16.514737 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 22:02:16.514863 24011 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:02:16.517037 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 23882
I20260620 22:02:17.308527 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:17.308213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:37225)
0620 22:02:17.308431 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:17.308468 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":57,"threads_started":1}
I20260620 22:02:17.387840 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:17.387636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:44413)
0620 22:02:17.387740 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:17.387791 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260620 22:02:17.515326 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:17.515133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:32771)
0620 22:02:17.515252 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:17.515283 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260620 22:02:17.806533 24018 raft_consensus.cc:482] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:17.806710 24018 raft_consensus.cc:504] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:02:17.807003 24018 leader_election.cc:288] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041), 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045)
Jun 20 22:02:17 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.130@KRBTEST.COM
Jun 20 22:02:17 dist-test-slave-gl2g krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.131@KRBTEST.COM
I20260620 22:02:17.820433 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:17.807239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:56513 (local address 127.6.241.131:36041)
0620 22:02:17.807392 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:17.807402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:17.807430 (+ 28us) server_negotiation.cc:407] Connection header received
0620 22:02:17.807885 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:17.807887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:17.807929 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:17.808101 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:17.808821 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:17.809680 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:17.810703 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:17.811071 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:17.811195 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:17.816940 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:17.816944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:17.816944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:17.816954 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:17.818766 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:17.819165 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:17.819167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:17.819168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:17.819275 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:17.819465 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:17.819466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:17.819466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:17.819780 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:17.819863 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:17.820194 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 22:02:17.820320 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260620 22:02:17.820433 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:17.807254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:57873 (local address 127.6.241.130:34045)
0620 22:02:17.807396 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:17.807406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:17.807440 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:02:17.807901 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:17.807903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:17.807939 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:17.808079 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:17.808871 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:17.809724 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:17.810696 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:17.811071 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:17.811195 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:17.816526 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:17.816530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:17.816530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:17.816546 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:17.818514 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:17.819061 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:17.819063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:17.819064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:17.819174 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:17.819475 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:17.819476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:17.819476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:17.819780 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:17.819863 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:17.820189 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 22:02:17.820320 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":47,"threads_started":1}
I20260620 22:02:17.820444 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:17.807138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:36041 (local address 127.6.241.129:56513)
0620 22:02:17.807209 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:17.807219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:17.807272 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:17.808127 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:17.808130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:17.808485 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:17.808686 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:17.808692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:17.809792 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:17.809793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:17.810552 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:17.810553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:17.811163 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:17.811164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:17.811306 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:17.814714 (+ 3408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:17.814718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:17.816817 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:17.818833 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:17.818834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:17.818835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:17.819099 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:17.819341 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:17.819341 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:17.819342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:17.819401 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:17.819851 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:17.819858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:17.820011 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 22:02:17.820119 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 22:02:17.820306 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":29}
I20260620 22:02:17.820452 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:17.807138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:34045 (local address 127.6.241.129:57873)
0620 22:02:17.807276 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:17.807284 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:17.807321 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:17.808127 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:17.808130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:17.808704 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:17.808761 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:17.808763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:17.809836 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:17.809837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:17.810553 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:17.810555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:17.811167 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:17.811168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:17.811302 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:17.814686 (+ 3384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:17.814697 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:17.816416 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:17.818637 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:17.818646 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:17.818651 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:17.818975 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:17.819242 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:17.819243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:17.819243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:17.819383 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:17.819851 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:17.819858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:17.820011 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 22:02:17.820119 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 22:02:17.820356 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":177,"thread_start_us":61,"threads_started":1}
W20260620 22:02:17.821390 23634 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
W20260620 22:02:17.821748 23632 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
I20260620 22:02:17.821801 23632 leader_election.cc:302] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 250124f3d4184342b7dafbb0b48831c7; no voters: 63c905d15cd946ba901971361636cec9, eeb4c1a2b64847d284ab66ffd2c457d1
I20260620 22:02:17.821890 24018 raft_consensus.cc:2726] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:02:18.309113 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:18.308987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:55929)
0620 22:02:18.309038 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:18.309073 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:02:18.388440 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:18.388255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:34125)
0620 22:02:18.388373 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:18.388406 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
I20260620 22:02:18.515878 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:18.515688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:39439)
0620 22:02:18.515803 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:18.515838 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260620 22:02:19.309840 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:19.309638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:52141)
0620 22:02:19.309760 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:19.309794 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260620 22:02:19.389034 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:19.388822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:51029)
0620 22:02:19.388924 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:19.388956 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
I20260620 22:02:19.516455 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:19.516270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:56447)
0620 22:02:19.516388 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:19.516421 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260620 22:02:19.882479 24027 raft_consensus.cc:482] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:19.882561 24027 raft_consensus.cc:504] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:02:19.882869 24027 leader_election.cc:288] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041), 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045)
W20260620 22:02:19.883175 23632 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
W20260620 22:02:19.883275 23634 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
I20260620 22:02:19.883308 23634 leader_election.cc:302] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 250124f3d4184342b7dafbb0b48831c7; no voters: 63c905d15cd946ba901971361636cec9, eeb4c1a2b64847d284ab66ffd2c457d1
I20260620 22:02:19.883399 24027 raft_consensus.cc:2726] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:02:20.310704 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:20.310395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:43699)
0620 22:02:20.310578 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:20.310648 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":117,"threads_started":1}
I20260620 22:02:20.389828 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:20.389565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:37717)
0620 22:02:20.389760 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:20.389795 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":51,"threads_started":1}
I20260620 22:02:20.517005 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:20.516819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:47265)
0620 22:02:20.516923 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:20.516957 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260620 22:02:21.311388 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:21.311155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:39439)
0620 22:02:21.311300 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:21.311338 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260620 22:02:21.390444 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:21.390250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:57415)
0620 22:02:21.390378 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:21.390411 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260620 22:02:21.517545 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:21.517377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:54887)
0620 22:02:21.517482 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:21.517510 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
I20260620 22:02:22.312094 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:22.311836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:46325)
0620 22:02:22.311995 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:22.312045 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260620 22:02:22.391209 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:22.390927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:33859)
0620 22:02:22.391091 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:22.391134 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":59,"threads_started":1}
I20260620 22:02:22.518350 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:22.518084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:59885)
0620 22:02:22.518253 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:22.518303 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260620 22:02:22.611189 24037 raft_consensus.cc:482] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:22.611311 24037 raft_consensus.cc:504] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:02:22.611500 24037 leader_election.cc:288] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041), 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045)
W20260620 22:02:22.611897 23632 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
W20260620 22:02:22.611980 23634 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
I20260620 22:02:22.612098 23634 leader_election.cc:302] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 250124f3d4184342b7dafbb0b48831c7; no voters: 63c905d15cd946ba901971361636cec9, eeb4c1a2b64847d284ab66ffd2c457d1
I20260620 22:02:22.612197 24037 raft_consensus.cc:2726] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:02:23.312814 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:23.312608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:56027)
0620 22:02:23.312728 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:23.312767 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260620 22:02:23.392040 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:23.391808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:52495)
0620 22:02:23.391948 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:23.391994 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260620 22:02:23.519094 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:23.518882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:40901)
0620 22:02:23.519007 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:23.519049 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260620 22:02:24.313462 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:24.313255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:57071)
0620 22:02:24.313384 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:24.313417 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260620 22:02:24.392743 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:24.392485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:36609)
0620 22:02:24.392628 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:24.392679 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260620 22:02:24.519824 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:24.519638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:54805)
0620 22:02:24.519756 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:24.519789 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260620 22:02:25.314265 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:25.313906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:52331)
0620 22:02:25.314103 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:25.314166 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":56,"threads_started":1}
I20260620 22:02:25.393355 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:25.393161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:51357)
0620 22:02:25.393277 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:25.393314 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260620 22:02:25.520511 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:25.520325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:38545)
0620 22:02:25.520444 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:25.520477 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260620 22:02:26.314944 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:26.314731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:33433)
0620 22:02:26.314864 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:26.314895 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260620 22:02:26.393980 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:26.393754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:46439)
0620 22:02:26.393871 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:26.393921 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260620 22:02:26.521148 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:26.520975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:52495)
0620 22:02:26.521085 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:26.521115 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260620 22:02:27.315630 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:27.315409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:46037)
0620 22:02:27.315546 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:27.315583 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260620 22:02:27.394732 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:27.394444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:44863)
0620 22:02:27.394644 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:27.394685 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260620 22:02:27.521716 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:27.521540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:49977)
0620 22:02:27.521648 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:27.521680 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260620 22:02:27.630811 24053 raft_consensus.cc:482] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:27.630909 24053 raft_consensus.cc:504] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:02:27.631114 24053 leader_election.cc:288] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041), 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045)
W20260620 22:02:27.631459 23634 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
W20260620 22:02:27.631546 23632 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
I20260620 22:02:27.631637 23632 leader_election.cc:302] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 250124f3d4184342b7dafbb0b48831c7; no voters: 63c905d15cd946ba901971361636cec9, eeb4c1a2b64847d284ab66ffd2c457d1
I20260620 22:02:27.631716 24053 raft_consensus.cc:2726] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:02:28.316303 24054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:28.316077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:40319)
0620 22:02:28.316219 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:28.316254 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 22:02:28.395366 24055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:28.395184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:56213)
0620 22:02:28.395294 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:28.395326 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260620 22:02:28.522480 24056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:28.522264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:47191)
0620 22:02:28.522402 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:28.522437 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260620 22:02:29.317076 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:29.316846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:36329)
0620 22:02:29.316978 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:29.317014 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260620 22:02:29.396039 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:29.395840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:55399)
0620 22:02:29.395958 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:29.395996 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260620 22:02:29.523204 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:29.522964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:38853)
0620 22:02:29.523082 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:29.523143 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260620 22:02:30.317703 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:30.317508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:41097)
0620 22:02:30.317623 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:30.317653 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260620 22:02:30.396708 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:30.396526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:36273)
0620 22:02:30.396635 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:30.396669 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260620 22:02:30.523823 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:30.523611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:57635)
0620 22:02:30.523745 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:30.523781 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260620 22:02:31.007923 24063 raft_consensus.cc:482] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:31.008011 24063 raft_consensus.cc:504] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb4c1a2b64847d284ab66ffd2c457d1" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 36041 } } peers { permanent_uuid: "63c905d15cd946ba901971361636cec9" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 34045 } } peers { permanent_uuid: "250124f3d4184342b7dafbb0b48831c7" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42537 } }
I20260620 22:02:31.008165 24063 leader_election.cc:288] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041), 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045)
W20260620 22:02:31.008524 23634 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeb4c1a2b64847d284ab66ffd2c457d1 (127.6.241.131:36041): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
W20260620 22:02:31.008639 23632 leader_election.cc:341] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63c905d15cd946ba901971361636cec9 (127.6.241.130:34045): Not found: Tablet not found: d64f3550d9be4874a6d2af743731d24e
I20260620 22:02:31.008697 23632 leader_election.cc:302] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 250124f3d4184342b7dafbb0b48831c7; no voters: 63c905d15cd946ba901971361636cec9, eeb4c1a2b64847d284ab66ffd2c457d1
I20260620 22:02:31.008786 24063 raft_consensus.cc:2726] T d64f3550d9be4874a6d2af743731d24e P 250124f3d4184342b7dafbb0b48831c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:02:31.318431 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:31.318214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.129:49289)
0620 22:02:31.318352 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:31.318387 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260620 22:02:31.397395 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:31.397192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.130:60719)
0620 22:02:31.397317 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:31.397352 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260620 22:02:31.517560 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 23581
I20260620 22:02:31.522243 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 23750
I20260620 22:02:31.524492 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:31.524297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41935 (local address 127.6.241.131:56395)
0620 22:02:31.524419 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:31.524455 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.190:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260620 22:02:31.526854 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 23882
2026-06-20T22:02:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58189 ms)
[----------] 30 tests from SecurityITest (233054 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:31 dist-test-slave-gl2g krb5kdc[24069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:31 dist-test-slave-gl2g krb5kdc[24069](info): set up 2 sockets
Jun 20 22:02:31 dist-test-slave-gl2g krb5kdc[24069](info): commencing operation
krb5kdc: starting...
W20260620 22:02:32.574348 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:32 dist-test-slave-gl2g krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:02:32.682858 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34427
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:34427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:32.760797 24082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:32.761021 24082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:32.761060 24082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:32.762243 24082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:32.762313 24082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:32.762346 24082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:32.762378 24082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:32.762403 24082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:32.763875 24082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:34427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:32.764149 24082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:32.764381 24082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:32.769448 24090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:32.769474 24082 server_base.cc:1034] running on GCE node
W20260620 22:02:32.769449 24087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:32.769456 24088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:32.769929 24082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:32.770224 24082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:32.771387 24082 hybrid_clock.cc:648] HybridClock initialized: now 1781992952771373 us; error 27 us; skew 500 ppm
Jun 20 22:02:32 dist-test-slave-gl2g krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992952, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:32.773741 24082 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:02:32.774197 24082 webserver.cc:460] Webserver started at http://127.6.241.190:33967/ using document root <none> and password file <none>
I20260620 22:02:32.774421 24082 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:32.774489 24082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:32.774647 24082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:32.775665 24082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "147b00c2eaa447728db1a49848296c74"
format_stamp: "Formatted at 2026-06-20 22:02:32 on dist-test-slave-gl2g"
server_key: "951ecb6a25b18864d56dd6b8af82dd73"
server_key_iv: "285aad6d1334271d163c80d3a27fc1e1"
server_key_version: "encryptionkey@0"
I20260620 22:02:32.776027 24082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "147b00c2eaa447728db1a49848296c74"
format_stamp: "Formatted at 2026-06-20 22:02:32 on dist-test-slave-gl2g"
server_key: "951ecb6a25b18864d56dd6b8af82dd73"
server_key_iv: "285aad6d1334271d163c80d3a27fc1e1"
server_key_version: "encryptionkey@0"
I20260620 22:02:32.777859 24082 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 22:02:32.779002 24097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:32.779198 24082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:32.779282 24082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "147b00c2eaa447728db1a49848296c74"
format_stamp: "Formatted at 2026-06-20 22:02:32 on dist-test-slave-gl2g"
server_key: "951ecb6a25b18864d56dd6b8af82dd73"
server_key_iv: "285aad6d1334271d163c80d3a27fc1e1"
server_key_version: "encryptionkey@0"
I20260620 22:02:32.779335 24082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:32.801504 24082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:32.804486 24082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:32.804661 24082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:32.809391 24082 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:34427
I20260620 22:02:32.809446 24149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:34427 every 8 connection(s)
I20260620 22:02:32.809803 24082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:32.810971 24150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:32.814682 24150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74: Bootstrap starting.
I20260620 22:02:32.815426 24150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:32.815759 24150 log.cc:824] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:32.816673 24150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74: No bootstrap required, opened a new log
I20260620 22:02:32.817289 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 24082
I20260620 22:02:32.817406 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:32.817618 7110 external_mini_cluster.cc:1428] Setting key bf34e1400f9ba24eff47fc9285a8f759
I20260620 22:02:32.819543 24150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147b00c2eaa447728db1a49848296c74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34427 } }
I20260620 22:02:32.819738 24150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:32.819780 24150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147b00c2eaa447728db1a49848296c74, State: Initialized, Role: FOLLOWER
I20260620 22:02:32.819978 24150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147b00c2eaa447728db1a49848296c74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34427 } }
I20260620 22:02:32.820079 24150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:32.820149 24150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:32.820230 24150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:32.821044 24150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147b00c2eaa447728db1a49848296c74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34427 } }
I20260620 22:02:32.821245 24150 leader_election.cc:302] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147b00c2eaa447728db1a49848296c74; no voters:
I20260620 22:02:32.821480 24150 leader_election.cc:288] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:32.821511 24155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:32.821746 24155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [term 1 LEADER]: Becoming Leader. State: Replica: 147b00c2eaa447728db1a49848296c74, State: Running, Role: LEADER
I20260620 22:02:32.821888 24155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147b00c2eaa447728db1a49848296c74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34427 } }
I20260620 22:02:32.821911 24150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:32.822526 24155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147b00c2eaa447728db1a49848296c74. Latest consensus state: current_term: 1 leader_uuid: "147b00c2eaa447728db1a49848296c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147b00c2eaa447728db1a49848296c74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34427 } } }
I20260620 22:02:32.822916 24155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:32.822849 24156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147b00c2eaa447728db1a49848296c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147b00c2eaa447728db1a49848296c74" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34427 } } }
I20260620 22:02:32.823299 24156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:32.823439 24164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:32.823935 24164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:32.825733 24164 catalog_manager.cc:1347] Generated new cluster ID: 13c4200a8803444c9932e5d40625aa3f
I20260620 22:02:32.825785 24164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:02:32 dist-test-slave-gl2g krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:32.830317 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:32.818267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.6.241.190:34427)
0620 22:02:32.818443 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:32.818450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:32.818474 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:02:32.819637 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:32.819644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:32.819694 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:32.819879 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:32.820717 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:32.822040 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:32.823211 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:32.823915 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:32.824013 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:32.826845 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:32.826849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:32.826850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:32.826860 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:32.828733 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:32.829144 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:32.829146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:32.829147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:32.829243 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:32.829450 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:32.829450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:32.829451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:32.829716 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:32.829855 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:32.830008 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 22:02:32.830141 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260620 22:02:32.835634 24164 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:32.836098 24164 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:32.838837 24164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74: Generated new TSK 0
I20260620 22:02:32.838991 24164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 22:02:32.863574 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34427
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:32.941108 24176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:32.941320 24176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:32.941355 24176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:32.942423 24176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:32.942487 24176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:32.942529 24176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:02:32.943936 24176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:32.944183 24176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:32.944397 24176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:32.949949 24182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:32.949976 24181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:32.949963 24184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:32.950188 24176 server_base.cc:1034] running on GCE node
I20260620 22:02:32.950371 24176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:32.950656 24176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:32.951800 24176 hybrid_clock.cc:648] HybridClock initialized: now 1781992952951784 us; error 27 us; skew 500 ppm
Jun 20 22:02:32 dist-test-slave-gl2g krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992952, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:32.954136 24176 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:02:32.954589 24176 webserver.cc:460] Webserver started at http://127.6.241.129:44061/ using document root <none> and password file <none>
I20260620 22:02:32.954852 24176 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:32.954923 24176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:32.955068 24176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:32.956056 24176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "2b579e25b741454988fd9f83b5af5f1f"
format_stamp: "Formatted at 2026-06-20 22:02:32 on dist-test-slave-gl2g"
server_key: "4d96f7b1df6315b361ab7a1284550b7c"
server_key_iv: "4c6f459ecbb5c25980dd09e2b368091a"
server_key_version: "encryptionkey@0"
I20260620 22:02:32.956421 24176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "2b579e25b741454988fd9f83b5af5f1f"
format_stamp: "Formatted at 2026-06-20 22:02:32 on dist-test-slave-gl2g"
server_key: "4d96f7b1df6315b361ab7a1284550b7c"
server_key_iv: "4c6f459ecbb5c25980dd09e2b368091a"
server_key_version: "encryptionkey@0"
I20260620 22:02:32.957747 24176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:02:32.958455 24191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:32.958665 24176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:32.958757 24176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "2b579e25b741454988fd9f83b5af5f1f"
format_stamp: "Formatted at 2026-06-20 22:02:32 on dist-test-slave-gl2g"
server_key: "4d96f7b1df6315b361ab7a1284550b7c"
server_key_iv: "4c6f459ecbb5c25980dd09e2b368091a"
server_key_version: "encryptionkey@0"
I20260620 22:02:32.958835 24176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:32.973250 24176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:32.975065 24176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:32.975217 24176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:32.975441 24176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:32.975734 24176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:32.975790 24176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:32.975831 24176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:32.975885 24176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:32.981776 24176 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:44519
I20260620 22:02:32.981825 24304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:44519 every 8 connection(s)
I20260620 22:02:32.982111 24176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:02:32 dist-test-slave-gl2g krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992952, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:32.987849 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 24176
I20260620 22:02:32.987993 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:02:32.988219 7110 external_mini_cluster.cc:1428] Setting key 67bcdd9bf5493f994b815038ae7f2156
I20260620 22:02:32.991247 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:32.982824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:48515 (local address 127.6.241.190:34427)
0620 22:02:32.982880 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:32.982882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:32.983089 (+ 207us) server_negotiation.cc:407] Connection header received
0620 22:02:32.983540 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:32.983541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:32.983551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:32.983601 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:32.984606 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:32.985109 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:32.986025 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:32.986219 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:32.986270 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:32.988367 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:32.988369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:32.988369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:32.988377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:32.989818 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:32.990229 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:32.990230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:32.990231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:32.990290 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:32.990542 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:32.990543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:32.990543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:32.990720 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:32.990794 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:32.991064 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 22:02:32.991150 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:02:32.991412 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:32.982808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:34427 (local address 127.6.241.129:48515)
0620 22:02:32.982968 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:32.983003 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:32.983470 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:32.983653 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:32.983656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:32.984318 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:32.984537 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:32.984542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:32.985186 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:32.985186 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:32.985950 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:32.985952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:32.986290 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:32.986291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:32.986395 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:32.986840 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:32.986851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:32.988263 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:32.989920 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:32.989927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:32.989929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:32.990147 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:32.990360 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:32.990361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:32.990361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:32.990473 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:32.990787 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:32.990793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:32.990928 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 22:02:32.991044 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 22:02:32.991188 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260620 22:02:32.991896 24305 heartbeater.cc:347] Connected to a master server at 127.6.241.190:34427
I20260620 22:02:32.991984 24305 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:32.992172 24305 heartbeater.cc:511] Master 127.6.241.190:34427 requested a full tablet report, sending...
I20260620 22:02:32.992739 24114 ts_manager.cc:194] Registered new tserver with Master: 2b579e25b741454988fd9f83b5af5f1f (127.6.241.129:44519)
I20260620 22:02:32.993714 24114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:48515
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 22:02:33.011015 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34427
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:33.089792 24310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:33.089977 24310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:33.089999 24310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:33.091074 24310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:33.091109 24310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:33.091133 24310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:02:33.092442 24310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:33.092602 24310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:33.092752 24310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:33.097379 24318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.097402 24315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:33.097452 24310 server_base.cc:1034] running on GCE node
W20260620 22:02:33.097421 24316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:33.097823 24310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:33.098064 24310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:33.099222 24310 hybrid_clock.cc:648] HybridClock initialized: now 1781992953099214 us; error 41 us; skew 500 ppm
Jun 20 22:02:33 dist-test-slave-gl2g krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992953, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:33.101541 24310 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:02:33.101966 24310 webserver.cc:460] Webserver started at http://127.6.241.130:42457/ using document root <none> and password file <none>
I20260620 22:02:33.102171 24310 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:33.102236 24310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:33.102378 24310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:33.103366 24310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "b591aba3a8f84212b0953f7ae1c993b2"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "a124fe0890b37afbf2fed23a70daa089"
server_key_iv: "0ceb1eb21464741965f2c3fd067ffdfe"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.103724 24310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "b591aba3a8f84212b0953f7ae1c993b2"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "a124fe0890b37afbf2fed23a70daa089"
server_key_iv: "0ceb1eb21464741965f2c3fd067ffdfe"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.104972 24310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:02:33.105677 24325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.105834 24310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:02:33.105932 24310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "b591aba3a8f84212b0953f7ae1c993b2"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "a124fe0890b37afbf2fed23a70daa089"
server_key_iv: "0ceb1eb21464741965f2c3fd067ffdfe"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.106009 24310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:33.119859 24310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:33.121613 24310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:33.121757 24310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:33.121971 24310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:33.122299 24310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:33.122363 24310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.122393 24310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:33.122435 24310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.128793 24310 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:41577
I20260620 22:02:33.128839 24438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:41577 every 8 connection(s)
I20260620 22:02:33.129106 24310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:02:33 dist-test-slave-gl2g krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992953, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:33.135243 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 24310
I20260620 22:02:33.135340 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:02:33.135527 7110 external_mini_cluster.cc:1428] Setting key 8b0ed422ba9950d1d8d4f8105af08aa3
I20260620 22:02:33.137952 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.129751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:34427 (local address 127.6.241.130:59693)
0620 22:02:33.129919 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.129950 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:33.130402 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:33.130668 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:33.130672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:33.131173 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:33.131343 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.131347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.132033 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.132035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.132781 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.132783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.133181 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.133182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.133294 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.133706 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:33.133713 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:33.134926 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:33.136597 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:33.136604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:33.136606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:33.136841 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:33.137071 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:33.137072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:33.137072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:33.137186 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:33.137493 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:33.137501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:33.137613 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 22:02:33.137705 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 22:02:33.137814 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260620 22:02:33.137951 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.129763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:59693 (local address 127.6.241.190:34427)
0620 22:02:33.129833 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.129835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.130033 (+ 198us) server_negotiation.cc:407] Connection header received
0620 22:02:33.130517 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.130518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.130529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.130584 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:33.131444 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.131943 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.132882 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.133085 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.133173 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.135003 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:33.135004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:33.135004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:33.135010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:33.136499 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:33.136939 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:33.136940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:33.136941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:33.137003 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:33.137257 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:33.137258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:33.137258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:33.137404 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:33.137478 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.137751 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.137834 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:02:33.138381 24439 heartbeater.cc:347] Connected to a master server at 127.6.241.190:34427
I20260620 22:02:33.138479 24439 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:33.138664 24439 heartbeater.cc:511] Master 127.6.241.190:34427 requested a full tablet report, sending...
I20260620 22:02:33.138999 24114 ts_manager.cc:194] Registered new tserver with Master: b591aba3a8f84212b0953f7ae1c993b2 (127.6.241.130:41577)
I20260620 22:02:33.139379 24114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:59693
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 22:02:33.158735 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34427
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:33.236788 24444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:33.236999 24444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:33.237022 24444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:33.238137 24444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:33.238173 24444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:33.238197 24444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:02:33.239638 24444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:34427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:33.239804 24444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:33.239960 24444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:33.244607 24452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.244648 24449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.244621 24450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:33.244791 24444 server_base.cc:1034] running on GCE node
I20260620 22:02:33.244968 24444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:33.245167 24444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:33.246313 24444 hybrid_clock.cc:648] HybridClock initialized: now 1781992953246304 us; error 32 us; skew 500 ppm
Jun 20 22:02:33 dist-test-slave-gl2g krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992953, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:33.248679 24444 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:02:33.249112 24444 webserver.cc:460] Webserver started at http://127.6.241.131:34929/ using document root <none> and password file <none>
I20260620 22:02:33.249329 24444 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:33.249395 24444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:33.249521 24444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:33.250525 24444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "1421e29db9dc4179a9f1603e36d73662"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "9380e4444468184b4eea2b8d451d2f1b"
server_key_iv: "325ecd8361eeefd795eaf5c7365f8a0b"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.250986 24444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "1421e29db9dc4179a9f1603e36d73662"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "9380e4444468184b4eea2b8d451d2f1b"
server_key_iv: "325ecd8361eeefd795eaf5c7365f8a0b"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.252259 24444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:33.252971 24459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.253136 24444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:33.253229 24444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "1421e29db9dc4179a9f1603e36d73662"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "9380e4444468184b4eea2b8d451d2f1b"
server_key_iv: "325ecd8361eeefd795eaf5c7365f8a0b"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.253311 24444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:33.264477 24444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:33.266134 24444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:33.266281 24444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:33.266492 24444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:33.266830 24444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:33.266887 24444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.266919 24444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:33.266966 24444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.272797 24444 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:45143
I20260620 22:02:33.272842 24572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:45143 every 8 connection(s)
I20260620 22:02:33.273116 24444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:02:33 dist-test-slave-gl2g krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992953, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:33.282006 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.273812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:46131 (local address 127.6.241.190:34427)
0620 22:02:33.273881 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.273883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.274086 (+ 203us) server_negotiation.cc:407] Connection header received
0620 22:02:33.274576 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.274577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.274589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.274679 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:33.275568 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.276091 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.277124 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.277339 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.277425 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.279475 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:33.279477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:33.279477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:33.279485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:33.280628 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:33.281038 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:33.281039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:33.281040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:33.281107 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:33.281358 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:33.281359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:33.281360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:33.281487 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:33.281559 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.281838 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.281915 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 22:02:33.282181 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.273801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:34427 (local address 127.6.241.131:46131)
0620 22:02:33.273975 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.274007 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:33.274470 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:33.274770 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:33.274773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:33.275289 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:33.275470 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.275474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.276215 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.276216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.277007 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.277009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.277436 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.277437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.277557 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.277968 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:33.277976 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:33.279378 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:33.280725 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:33.280732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:33.280734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:33.280947 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:33.281175 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:33.281176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:33.281176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:33.281279 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:33.281550 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:33.281556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:33.281708 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 22:02:33.281814 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 22:02:33.281962 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260620 22:02:33.282717 24573 heartbeater.cc:347] Connected to a master server at 127.6.241.190:34427
I20260620 22:02:33.282819 24573 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:33.282801 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 24444
I20260620 22:02:33.282893 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:02:33.283003 24573 heartbeater.cc:511] Master 127.6.241.190:34427 requested a full tablet report, sending...
I20260620 22:02:33.283118 7110 external_mini_cluster.cc:1428] Setting key b9aace6e6e42326164c001a76f370531
I20260620 22:02:33.283427 24114 ts_manager.cc:194] Registered new tserver with Master: 1421e29db9dc4179a9f1603e36d73662 (127.6.241.131:45143)
I20260620 22:02:33.283897 24114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:46131
I20260620 22:02:33.284096 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:02:33.291100 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.285543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.6.241.190:34427)
0620 22:02:33.285593 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.285595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.285729 (+ 134us) server_negotiation.cc:407] Connection header received
0620 22:02:33.285763 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.285763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.285771 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.285816 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:33.286292 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.286800 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.287479 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.287645 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.287691 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.288714 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:33.288715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:33.288715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:33.288727 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:33.290018 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:33.290438 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:33.290438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:33.290439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:33.290483 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:33.290677 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:33.290678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:33.290678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:33.290812 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:33.290885 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.290988 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.291044 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:02:33.292557 24114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:02:33.294086 24114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:02:33.300544 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.296495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.6.241.129:44519)
0620 22:02:33.296652 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.296656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.296937 (+ 281us) server_negotiation.cc:407] Connection header received
0620 22:02:33.296988 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.296990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.297006 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.297078 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:33.297309 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.298024 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.299626 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.300111 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.300211 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.300274 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.300423 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.300470 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260620 22:02:33.300535 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.296336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:44519 (local address 127.0.0.1:44426)
0620 22:02:33.296836 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.296845 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:33.296894 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:33.297178 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:33.297180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:33.297182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:33.297228 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.297230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.298134 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.298135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.299525 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.299528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.300215 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.300216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.300294 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.300307 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:02:33.300406 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 22:02:33.300462 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":18,"thread_start_us":76,"threads_started":1}
I20260620 22:02:33.300729 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.296284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.6.241.131:45143)
0620 22:02:33.296556 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.296560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.296747 (+ 187us) server_negotiation.cc:407] Connection header received
0620 22:02:33.296909 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.296921 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.296948 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.297043 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:33.297191 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.298180 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.299606 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.300255 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.300310 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.300355 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.300584 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.300635 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":84,"threads_started":1}
I20260620 22:02:33.300904 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.296135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:45143 (local address 127.0.0.1:40746)
0620 22:02:33.296667 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.296676 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:33.296753 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:33.297045 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:33.297046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:33.297048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:33.297095 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.297096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.298278 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.298279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.299526 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.299528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.300383 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.300383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.300435 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.300441 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:02:33.300489 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:02:33.300542 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":33,"thread_start_us":60,"threads_started":1}
I20260620 22:02:33.301105 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.296129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:41577 (local address 127.0.0.1:58166)
0620 22:02:33.296427 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.296470 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:33.296551 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:33.296790 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:33.296792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:33.296795 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:33.296887 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.296889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.298230 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.298231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.299380 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.299383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.300369 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.300371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.300435 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.300439 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:02:33.300489 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:02:33.300542 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":58,"thread_start_us":54,"threads_started":1}
I20260620 22:02:33.301116 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.296245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.6.241.130:41577)
0620 22:02:33.296469 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.296473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.296549 (+ 76us) server_negotiation.cc:407] Connection header received
0620 22:02:33.296633 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.296635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.296651 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.296783 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:33.296975 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.298116 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.299475 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.300191 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.300265 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.300321 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.300984 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.301040 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260620 22:02:33.301338 24239 tablet_service.cc:1467] Processing CreateTablet for tablet 15246ed5af12448b885b7eacbe6e271a (DEFAULT_TABLE table=test-table [id=4f74ad52df3447d3be8bbd877e5103f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:33.301721 24239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15246ed5af12448b885b7eacbe6e271a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:33.303270 24373 tablet_service.cc:1467] Processing CreateTablet for tablet 15246ed5af12448b885b7eacbe6e271a (DEFAULT_TABLE table=test-table [id=4f74ad52df3447d3be8bbd877e5103f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:33.303270 24507 tablet_service.cc:1467] Processing CreateTablet for tablet 15246ed5af12448b885b7eacbe6e271a (DEFAULT_TABLE table=test-table [id=4f74ad52df3447d3be8bbd877e5103f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:33.303591 24507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15246ed5af12448b885b7eacbe6e271a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:33.303591 24373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15246ed5af12448b885b7eacbe6e271a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:33.304697 24587 tablet_bootstrap.cc:492] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f: Bootstrap starting.
I20260620 22:02:33.305282 24587 tablet_bootstrap.cc:654] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:33.305822 24587 log.cc:824] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:33.306527 24587 tablet_bootstrap.cc:492] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f: No bootstrap required, opened a new log
I20260620 22:02:33.306628 24587 ts_tablet_manager.cc:1397] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:02:33.306898 24589 tablet_bootstrap.cc:492] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: Bootstrap starting.
I20260620 22:02:33.307422 24589 tablet_bootstrap.cc:654] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:33.308001 24589 log.cc:824] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:33.308362 24590 tablet_bootstrap.cc:492] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: Bootstrap starting.
I20260620 22:02:33.308480 24587 raft_consensus.cc:348] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } }
I20260620 22:02:33.308631 24587 raft_consensus.cc:374] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:33.308682 24587 raft_consensus.cc:729] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b579e25b741454988fd9f83b5af5f1f, State: Initialized, Role: FOLLOWER
I20260620 22:02:33.308794 24587 consensus_queue.cc:260] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } }
I20260620 22:02:33.309005 24590 tablet_bootstrap.cc:654] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:33.309082 24587 ts_tablet_manager.cc:1428] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:02:33.309120 24305 heartbeater.cc:503] Master 127.6.241.190:34427 was elected leader, sending a full tablet report...
I20260620 22:02:33.309526 24590 log.cc:824] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:33.310094 24589 tablet_bootstrap.cc:492] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: No bootstrap required, opened a new log
I20260620 22:02:33.310165 24589 ts_tablet_manager.cc:1397] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 22:02:33.310271 24590 tablet_bootstrap.cc:492] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: No bootstrap required, opened a new log
I20260620 22:02:33.310330 24590 ts_tablet_manager.cc:1397] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:02:33.312229 24590 raft_consensus.cc:348] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } }
I20260620 22:02:33.312191 24589 raft_consensus.cc:348] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } }
I20260620 22:02:33.312368 24589 raft_consensus.cc:374] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:33.312368 24590 raft_consensus.cc:374] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:33.312404 24590 raft_consensus.cc:729] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1421e29db9dc4179a9f1603e36d73662, State: Initialized, Role: FOLLOWER
I20260620 22:02:33.312404 24589 raft_consensus.cc:729] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b591aba3a8f84212b0953f7ae1c993b2, State: Initialized, Role: FOLLOWER
I20260620 22:02:33.312511 24589 consensus_queue.cc:260] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } }
I20260620 22:02:33.312511 24590 consensus_queue.cc:260] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } }
I20260620 22:02:33.312777 24590 ts_tablet_manager.cc:1428] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:02:33.312777 24589 ts_tablet_manager.cc:1428] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:02:33.312811 24439 heartbeater.cc:503] Master 127.6.241.190:34427 was elected leader, sending a full tablet report...
I20260620 22:02:33.312892 24573 heartbeater.cc:503] Master 127.6.241.190:34427 was elected leader, sending a full tablet report...
I20260620 22:02:33.355057 24592 raft_consensus.cc:482] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:33.355163 24592 raft_consensus.cc:504] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } }
I20260620 22:02:33.355456 24592 leader_election.cc:288] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b591aba3a8f84212b0953f7ae1c993b2 (127.6.241.130:41577), 1421e29db9dc4179a9f1603e36d73662 (127.6.241.131:45143)
I20260620 22:02:33.358379 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.355646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:41577 (local address 127.6.241.129:60265)
0620 22:02:33.355685 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.355697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:33.355733 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:33.355931 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:33.355932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:33.355936 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:33.356046 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.356048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.356721 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.356722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.357518 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.357520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.358095 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.358096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.358174 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.358177 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:33.358227 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:02:33.358256 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:02:33.358407 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.355733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:60265 (local address 127.6.241.130:41577)
0620 22:02:33.355803 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.355805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.355812 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:02:33.355839 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.355840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.355852 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.355900 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:33.356105 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.356630 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.357612 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.358008 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.358063 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.358070 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.358257 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.358314 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:02:33.358461 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.355728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:50407 (local address 127.6.241.131:45143)
0620 22:02:33.355809 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.355811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.355965 (+ 154us) server_negotiation.cc:407] Connection header received
0620 22:02:33.355998 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.355999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.356011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.356078 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:33.356226 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.356690 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.357659 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.358022 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.358113 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.358121 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.358321 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.358388 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 22:02:33.358713 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.355647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:45143 (local address 127.6.241.129:50407)
0620 22:02:33.355901 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.355910 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:33.355957 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:33.356087 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:33.356088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:33.356089 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:33.356157 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.356159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.356771 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.356772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.357547 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.357549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.358128 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.358129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.358207 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.358210 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:33.358303 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 22:02:33.358334 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":60,"thread_start_us":148,"threads_started":1}
I20260620 22:02:33.358739 24393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15246ed5af12448b885b7eacbe6e271a" candidate_uuid: "2b579e25b741454988fd9f83b5af5f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b591aba3a8f84212b0953f7ae1c993b2" is_pre_election: true
I20260620 22:02:33.358894 24393 raft_consensus.cc:2454] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b579e25b741454988fd9f83b5af5f1f in term 0.
I20260620 22:02:33.359071 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15246ed5af12448b885b7eacbe6e271a" candidate_uuid: "2b579e25b741454988fd9f83b5af5f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1421e29db9dc4179a9f1603e36d73662" is_pre_election: true
I20260620 22:02:33.359184 24527 raft_consensus.cc:2454] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b579e25b741454988fd9f83b5af5f1f in term 0.
I20260620 22:02:33.359154 24195 leader_election.cc:302] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b579e25b741454988fd9f83b5af5f1f, b591aba3a8f84212b0953f7ae1c993b2; no voters:
I20260620 22:02:33.359299 24592 raft_consensus.cc:2781] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:02:33.359359 24592 raft_consensus.cc:482] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:02:33.359385 24592 raft_consensus.cc:3037] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:33.360288 24592 raft_consensus.cc:504] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } }
I20260620 22:02:33.360409 24592 leader_election.cc:288] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [CANDIDATE]: Term 1 election: Requested vote from peers b591aba3a8f84212b0953f7ae1c993b2 (127.6.241.130:41577), 1421e29db9dc4179a9f1603e36d73662 (127.6.241.131:45143)
I20260620 22:02:33.366077 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15246ed5af12448b885b7eacbe6e271a" candidate_uuid: "2b579e25b741454988fd9f83b5af5f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1421e29db9dc4179a9f1603e36d73662"
I20260620 22:02:33.366104 24393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15246ed5af12448b885b7eacbe6e271a" candidate_uuid: "2b579e25b741454988fd9f83b5af5f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b591aba3a8f84212b0953f7ae1c993b2"
I20260620 22:02:33.366165 24527 raft_consensus.cc:3037] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:33.366165 24393 raft_consensus.cc:3037] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:33.367095 24527 raft_consensus.cc:2454] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b579e25b741454988fd9f83b5af5f1f in term 1.
I20260620 22:02:33.367095 24393 raft_consensus.cc:2454] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b579e25b741454988fd9f83b5af5f1f in term 1.
I20260620 22:02:33.367281 24194 leader_election.cc:302] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1421e29db9dc4179a9f1603e36d73662, 2b579e25b741454988fd9f83b5af5f1f; no voters:
I20260620 22:02:33.367380 24592 raft_consensus.cc:2781] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:33.367511 24592 raft_consensus.cc:686] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 1 LEADER]: Becoming Leader. State: Replica: 2b579e25b741454988fd9f83b5af5f1f, State: Running, Role: LEADER
I20260620 22:02:33.367597 24592 consensus_queue.cc:237] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } }
I20260620 22:02:33.368363 24114 catalog_manager.cc:5495] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b579e25b741454988fd9f83b5af5f1f (127.6.241.129). New cstate: current_term: 1 leader_uuid: "2b579e25b741454988fd9f83b5af5f1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b579e25b741454988fd9f83b5af5f1f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44519 } health_report { overall_health: HEALTHY } } }
W20260620 22:02:33.379628 24440 tablet.cc:2369] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:02:33.391378 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.388935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.6.241.129:44519)
0620 22:02:33.388996 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.388998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.389007 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:02:33.389073 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.389074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.389090 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.389164 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:02:33.389315 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.389784 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.390521 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.390774 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.390847 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.391023 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:02:33.391136 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:02:33.391223 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.391275 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.391298 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:02:33.393121 24527 raft_consensus.cc:1261] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [term 1 FOLLOWER]: Refusing update from remote peer 2b579e25b741454988fd9f83b5af5f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:33.393121 24393 raft_consensus.cc:1261] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [term 1 FOLLOWER]: Refusing update from remote peer 2b579e25b741454988fd9f83b5af5f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:33.393487 24597 consensus_queue.cc:1035] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b591aba3a8f84212b0953f7ae1c993b2" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:33.393616 24592 consensus_queue.cc:1035] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1421e29db9dc4179a9f1603e36d73662" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 45143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:33.396085 24602 mvcc.cc:205] Tried to move back new op lower bound from 7299043137095532544 to 7299043136994205696. Current Snapshot: MvccSnapshot[applied={T|T < 7299043137095532544}]
I20260620 22:02:33.396582 24601 mvcc.cc:205] Tried to move back new op lower bound from 7299043137095532544 to 7299043136994205696. Current Snapshot: MvccSnapshot[applied={T|T < 7299043137095532544}]
I20260620 22:02:33.397742 24114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:33.397841 24114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:33.398573 24114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 147b00c2eaa447728db1a49848296c74: Sending DeleteTablet for 3 replicas of tablet 15246ed5af12448b885b7eacbe6e271a
I20260620 22:02:33.398963 24507 tablet_service.cc:1514] Processing DeleteTablet for tablet 15246ed5af12448b885b7eacbe6e271a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:33 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:40746
I20260620 22:02:33.399166 24239 tablet_service.cc:1514] Processing DeleteTablet for tablet 15246ed5af12448b885b7eacbe6e271a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:33 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:44426
I20260620 22:02:33.399210 24373 tablet_service.cc:1514] Processing DeleteTablet for tablet 15246ed5af12448b885b7eacbe6e271a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:33 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:58166
I20260620 22:02:33.399408 24610 tablet_replica.cc:330] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f: stopping tablet replica
I20260620 22:02:33.399484 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 24176
I20260620 22:02:33.399506 24610 raft_consensus.cc:2229] T 15246ed5af12448b885b7eacbe6e271a P 2b579e25b741454988fd9f83b5af5f1f [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:02:33.400274 24611 tablet_replica.cc:330] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: stopping tablet replica
I20260620 22:02:33.400375 24611 raft_consensus.cc:2229] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:02:33.400458 24611 raft_consensus.cc:2258] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:33.400892 24611 ts_tablet_manager.cc:1905] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:33.401408 24609 tablet_replica.cc:330] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: stopping tablet replica
I20260620 22:02:33.401520 24609 raft_consensus.cc:2229] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:02:33.401610 24609 raft_consensus.cc:2258] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:33.402074 24609 ts_tablet_manager.cc:1905] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:33.402730 24611 ts_tablet_manager.cc:1918] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:33.402787 24611 log.cc:1196] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/15246ed5af12448b885b7eacbe6e271a
I20260620 22:02:33.403046 24611 ts_tablet_manager.cc:1939] T 15246ed5af12448b885b7eacbe6e271a P b591aba3a8f84212b0953f7ae1c993b2: Deleting consensus metadata
I20260620 22:02:33.403398 24101 catalog_manager.cc:4841] TS b591aba3a8f84212b0953f7ae1c993b2 (127.6.241.130:41577): tablet 15246ed5af12448b885b7eacbe6e271a (table test-table [id=4f74ad52df3447d3be8bbd877e5103f3]) successfully deleted
I20260620 22:02:33.404117 24609 ts_tablet_manager.cc:1918] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:33.404192 24609 log.cc:1196] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/15246ed5af12448b885b7eacbe6e271a
I20260620 22:02:33.404443 24609 ts_tablet_manager.cc:1939] T 15246ed5af12448b885b7eacbe6e271a P 1421e29db9dc4179a9f1603e36d73662: Deleting consensus metadata
I20260620 22:02:33.404804 24100 catalog_manager.cc:4841] TS 1421e29db9dc4179a9f1603e36d73662 (127.6.241.131:45143): tablet 15246ed5af12448b885b7eacbe6e271a (table test-table [id=4f74ad52df3447d3be8bbd877e5103f3]) successfully deleted
W20260620 22:02:33.405731 24100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:44519 (error 108)
I20260620 22:02:33.406181 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.406076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:44519 (local address 127.0.0.1:44446)
0620 22:02:33.406125 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.406157 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:02:33.406260 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 24310
W20260620 22:02:33.406388 24100 catalog_manager.cc:4584] TS 2b579e25b741454988fd9f83b5af5f1f (127.6.241.129:44519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15246ed5af12448b885b7eacbe6e271a: Network error: Client connection negotiation failed: client connection to 127.6.241.129:44519: connect: Connection refused (error 111)
I20260620 22:02:33.411041 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 24444
I20260620 22:02:33.415446 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 24082
2026-06-20T22:02:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1888 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T22:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:33Z Disabled control of system clock
I20260620 22:02:33.441546 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41651
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:41651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:02:33.519172 24616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:33.519377 24616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:33.519397 24616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:33.520488 24616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:33.520524 24616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:33.520534 24616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:33.520546 24616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:33.520596 24616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:33.521816 24616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:41651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:41651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:33.522042 24616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:33.522310 24616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:33.526721 24621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.526727 24624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.526738 24622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:33.526958 24616 server_base.cc:1034] running on GCE node
I20260620 22:02:33.527205 24616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:33.527493 24616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:33.528626 24616 hybrid_clock.cc:648] HybridClock initialized: now 1781992953528617 us; error 26 us; skew 500 ppm
I20260620 22:02:33.529747 24616 webserver.cc:460] Webserver started at http://127.6.241.190:41317/ using document root <none> and password file <none>
I20260620 22:02:33.529986 24616 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:33.530032 24616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:33.530179 24616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:33.531211 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "788307795c24466197dc71c36cdb4154"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "54bdd3f55e7eb957b85f51b34651e8c5"
server_key_iv: "bdd3b1e68e49bc8239ea5339bca14ffe"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.531563 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "788307795c24466197dc71c36cdb4154"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "54bdd3f55e7eb957b85f51b34651e8c5"
server_key_iv: "bdd3b1e68e49bc8239ea5339bca14ffe"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.532833 24616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:02:33.533504 24630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.533663 24616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:33.533759 24616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "788307795c24466197dc71c36cdb4154"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "54bdd3f55e7eb957b85f51b34651e8c5"
server_key_iv: "bdd3b1e68e49bc8239ea5339bca14ffe"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.533835 24616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:33.566371 24616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:33.566766 24616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:33.566927 24616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:33.571014 24616 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:41651
I20260620 22:02:33.571074 24682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:41651 every 8 connection(s)
I20260620 22:02:33.571436 24616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:33.572402 24683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:33.574748 24683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154: Bootstrap starting.
I20260620 22:02:33.575287 24683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:33.575531 24683 log.cc:824] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:33.575785 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 24616
I20260620 22:02:33.575893 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:33.576095 7110 external_mini_cluster.cc:1428] Setting key 7e97f9df7454937d92757b996c7bc2ef
I20260620 22:02:33.576355 24683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154: No bootstrap required, opened a new log
I20260620 22:02:33.578305 24683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788307795c24466197dc71c36cdb4154" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41651 } }
I20260620 22:02:33.578439 24683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:33.578462 24683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788307795c24466197dc71c36cdb4154, State: Initialized, Role: FOLLOWER
I20260620 22:02:33.578631 24683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788307795c24466197dc71c36cdb4154" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41651 } }
I20260620 22:02:33.578727 24683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:33.578769 24683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:33.578817 24683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:33.579442 24683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788307795c24466197dc71c36cdb4154" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41651 } }
I20260620 22:02:33.579525 24683 leader_election.cc:302] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788307795c24466197dc71c36cdb4154; no voters:
I20260620 22:02:33.579710 24683 leader_election.cc:288] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:33.579742 24688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:33.579846 24688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [term 1 LEADER]: Becoming Leader. State: Replica: 788307795c24466197dc71c36cdb4154, State: Running, Role: LEADER
I20260620 22:02:33.579924 24688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788307795c24466197dc71c36cdb4154" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41651 } }
I20260620 22:02:33.579967 24683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:33.580235 24688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788307795c24466197dc71c36cdb4154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788307795c24466197dc71c36cdb4154" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41651 } } }
I20260620 22:02:33.580303 24688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:33.580439 24688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788307795c24466197dc71c36cdb4154. Latest consensus state: current_term: 1 leader_uuid: "788307795c24466197dc71c36cdb4154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788307795c24466197dc71c36cdb4154" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 41651 } } }
I20260620 22:02:33.580500 24688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:33.581096 24693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:33.581557 24693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:33.583367 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.576567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.6.241.190:41651)
0620 22:02:33.576823 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.576833 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.576872 (+ 39us) server_negotiation.cc:407] Connection header received
0620 22:02:33.577644 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.577652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.577710 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.577867 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:33.578076 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.579525 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.580693 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.581361 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.581511 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.582743 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:33.582746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:33.582747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:33.582931 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:33.582961 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:33.582971 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:33.583050 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.583126 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.583201 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260620 22:02:33.586812 24693 catalog_manager.cc:1347] Generated new cluster ID: 5265f8dce0c5466b8980bf57bf4f44a2
I20260620 22:02:33.586886 24693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:33.592454 24693 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:33.593111 24693 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:33.601365 24693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154: Generated new TSK 0
I20260620 22:02:33.601511 24693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:33.611320 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41651
--builtin_ntp_servers=127.6.241.148:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:02:33.689406 24707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:33.689620 24707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:33.689654 24707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:33.690824 24707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:33.690893 24707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:33.690950 24707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:02:33.692312 24707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:33.692557 24707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:33.692834 24707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:33.698552 24715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.698549 24713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:33.698818 24707 server_base.cc:1034] running on GCE node
W20260620 22:02:33.698535 24712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:33.699080 24707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:33.699419 24707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:33.700601 24707 hybrid_clock.cc:648] HybridClock initialized: now 1781992953700578 us; error 46 us; skew 500 ppm
I20260620 22:02:33.701925 24707 webserver.cc:460] Webserver started at http://127.6.241.129:40263/ using document root <none> and password file <none>
I20260620 22:02:33.702172 24707 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:33.702241 24707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:33.702363 24707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:33.703680 24707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "e3b8245d7e214f618b56dd3ec6d61e5f"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "b922dd4d54d2f66b66d0fd03b64f29a6"
server_key_iv: "29df2627f4d89744812f41899efb3d59"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.704044 24707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "e3b8245d7e214f618b56dd3ec6d61e5f"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "b922dd4d54d2f66b66d0fd03b64f29a6"
server_key_iv: "29df2627f4d89744812f41899efb3d59"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.705776 24707 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 22:02:33.706930 24721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.707103 24707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:02:33.707175 24707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "e3b8245d7e214f618b56dd3ec6d61e5f"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "b922dd4d54d2f66b66d0fd03b64f29a6"
server_key_iv: "29df2627f4d89744812f41899efb3d59"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.707249 24707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:33.732690 24707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:33.732968 24707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:33.733119 24707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:33.733336 24707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:33.733652 24707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:33.733700 24707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.733747 24707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:33.733774 24707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.739552 24707 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:38925
I20260620 22:02:33.739622 24834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:38925 every 8 connection(s)
I20260620 22:02:33.739919 24707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:02:33.744916 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.740599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:53047 (local address 127.6.241.190:41651)
0620 22:02:33.740637 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.740640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.740844 (+ 204us) server_negotiation.cc:407] Connection header received
0620 22:02:33.741367 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.741368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.741385 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.741449 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:33.742098 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.742636 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.743591 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.743801 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.743884 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.744631 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:33.744633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:33.744633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:33.744638 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:33.744651 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:33.744654 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:33.744706 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.744816 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.744842 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:02:33.745113 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.740578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41651 (local address 127.6.241.129:53047)
0620 22:02:33.740733 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.740763 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:33.741256 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:33.741512 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:33.741515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:33.741524 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:33.742006 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.742011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.742713 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.742714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.743488 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.743490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.743894 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.743895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.744031 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.744478 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:02:33.744478 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:33.744503 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:02:33.744540 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:33.744728 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:33.744732 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:33.744735 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:33.744819 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:02:33.744912 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260620 22:02:33.745615 24835 heartbeater.cc:347] Connected to a master server at 127.6.241.190:41651
I20260620 22:02:33.745683 24835 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:33.745815 24835 heartbeater.cc:511] Master 127.6.241.190:41651 requested a full tablet report, sending...
I20260620 22:02:33.746243 24647 ts_manager.cc:194] Registered new tserver with Master: e3b8245d7e214f618b56dd3ec6d61e5f (127.6.241.129:38925)
I20260620 22:02:33.746479 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 24707
I20260620 22:02:33.746568 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:02:33.746798 7110 external_mini_cluster.cc:1428] Setting key 9308f7677ef8dc414cfad7299c65038c
I20260620 22:02:33.747149 24647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.129:53047
I20260620 22:02:33.747794 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41651
--builtin_ntp_servers=127.6.241.148:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:02:33.827519 24838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:33.827699 24838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:33.827720 24838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:33.828850 24838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:33.828889 24838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:33.828913 24838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:02:33.830214 24838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:33.830444 24838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:33.830768 24838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:33.835382 24846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.835398 24843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.835443 24844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:33.835584 24838 server_base.cc:1034] running on GCE node
I20260620 22:02:33.835809 24838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:33.836050 24838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:33.837203 24838 hybrid_clock.cc:648] HybridClock initialized: now 1781992953837191 us; error 42 us; skew 500 ppm
I20260620 22:02:33.838361 24838 webserver.cc:460] Webserver started at http://127.6.241.130:39669/ using document root <none> and password file <none>
I20260620 22:02:33.838691 24838 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:33.838759 24838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:33.838896 24838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:33.839864 24838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "57d9b825939248d3b57e06c8adcd1416"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "fd88f2e617f4fb760be07e9d453bc829"
server_key_iv: "9316ec2e8a40dda47b0ba8c641e1632c"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.840204 24838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "57d9b825939248d3b57e06c8adcd1416"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "fd88f2e617f4fb760be07e9d453bc829"
server_key_iv: "9316ec2e8a40dda47b0ba8c641e1632c"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.841395 24838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:02:33.842074 24852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.842258 24838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:02:33.842351 24838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "57d9b825939248d3b57e06c8adcd1416"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "fd88f2e617f4fb760be07e9d453bc829"
server_key_iv: "9316ec2e8a40dda47b0ba8c641e1632c"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.842437 24838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:33.884615 24838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:33.884939 24838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:33.885090 24838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:33.885324 24838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:33.885658 24838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:33.885713 24838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.885747 24838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:33.885776 24838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.891929 24838 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:35423
I20260620 22:02:33.891966 24965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:35423 every 8 connection(s)
I20260620 22:02:33.892318 24838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:02:33.892791 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 24838
I20260620 22:02:33.892891 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:02:33.893076 7110 external_mini_cluster.cc:1428] Setting key d7a2d8cc3dded15c21ca54b76f11e203
I20260620 22:02:33.894126 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41651
--builtin_ntp_servers=127.6.241.148:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:02:33.898372 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.893645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:37259 (local address 127.6.241.190:41651)
0620 22:02:33.893717 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:33.893719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:33.893964 (+ 245us) server_negotiation.cc:407] Connection header received
0620 22:02:33.894567 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:33.894569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:33.894581 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:33.894665 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:33.895364 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.895851 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.896868 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.897069 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.897154 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.898056 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:33.898058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:33.898058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:33.898063 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:33.898083 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:33.898087 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:33.898147 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:33.898264 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:02:33.898292 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:02:33.898651 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:33.893637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41651 (local address 127.6.241.130:37259)
0620 22:02:33.893837 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:33.893875 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:33.894470 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:33.894726 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:33.894730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:33.894737 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:33.895262 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.895267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.895943 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.895944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.896763 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:33.896765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:33.897173 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:33.897174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:33.897308 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:33.897907 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:02:33.897908 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:33.897935 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:02:33.897957 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:33.898149 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:33.898155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:33.898158 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:33.898264 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 22:02:33.898365 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260620 22:02:33.899051 24966 heartbeater.cc:347] Connected to a master server at 127.6.241.190:41651
I20260620 22:02:33.899139 24966 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:33.899340 24966 heartbeater.cc:511] Master 127.6.241.190:41651 requested a full tablet report, sending...
I20260620 22:02:33.899715 24646 ts_manager.cc:194] Registered new tserver with Master: 57d9b825939248d3b57e06c8adcd1416 (127.6.241.130:35423)
I20260620 22:02:33.900040 24646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.130:37259
W20260620 22:02:33.971926 24969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:33.972137 24969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:33.972168 24969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:33.973258 24969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:33.973316 24969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:33.973356 24969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:02:33.974558 24969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:41651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:33.974807 24969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:33.975059 24969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:33.979769 24974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.979739 24975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:33.979810 24977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:33.979821 24969 server_base.cc:1034] running on GCE node
I20260620 22:02:33.980142 24969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:33.980381 24969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:33.981565 24969 hybrid_clock.cc:648] HybridClock initialized: now 1781992953981562 us; error 46 us; skew 500 ppm
I20260620 22:02:33.982705 24969 webserver.cc:460] Webserver started at http://127.6.241.131:32787/ using document root <none> and password file <none>
I20260620 22:02:33.982949 24969 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:33.982992 24969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:33.983129 24969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:33.984139 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "7c6ddfa4242042bdac12e5155a19a405"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "121c2ebc7e5f70b8f70d7ff1d3ad8ca1"
server_key_iv: "6c9b865c449352549eb09003e9e28929"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.984536 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "7c6ddfa4242042bdac12e5155a19a405"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "121c2ebc7e5f70b8f70d7ff1d3ad8ca1"
server_key_iv: "6c9b865c449352549eb09003e9e28929"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.985819 24969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:33.986482 24983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.986673 24969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:33.986779 24969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "7c6ddfa4242042bdac12e5155a19a405"
format_stamp: "Formatted at 2026-06-20 22:02:33 on dist-test-slave-gl2g"
server_key: "121c2ebc7e5f70b8f70d7ff1d3ad8ca1"
server_key_iv: "6c9b865c449352549eb09003e9e28929"
server_key_version: "encryptionkey@0"
I20260620 22:02:33.986862 24969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:33.997987 24969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:33.998237 24969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:33.998376 24969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:33.998574 24969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:33.998919 24969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:33.998972 24969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:33.999010 24969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:33.999038 24969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:34.004617 24969 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:38283
I20260620 22:02:34.004653 25096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:38283 every 8 connection(s)
I20260620 22:02:34.004948 24969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:02:34.008617 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 24969
I20260620 22:02:34.008690 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:02:34.008870 7110 external_mini_cluster.cc:1428] Setting key 3836049654755a92dd2755dbf987a68b
I20260620 22:02:34.010085 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.005632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:42473 (local address 127.6.241.190:41651)
0620 22:02:34.005715 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:34.005718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:34.005913 (+ 195us) server_negotiation.cc:407] Connection header received
0620 22:02:34.006403 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:34.006405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:34.006418 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:34.006462 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:34.007181 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.007672 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.008628 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.008882 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.008958 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.009850 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:34.009850 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:34.009851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:34.009854 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:34.009868 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:34.009871 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:34.009939 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:34.010011 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 22:02:34.010031 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:02:34.010270 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.005593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:41651 (local address 127.6.241.131:42473)
0620 22:02:34.005800 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:34.005830 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:34.006286 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:34.006538 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:34.006542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:34.006551 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:34.007086 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.007091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.007771 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.007772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.008532 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.008533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.009001 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.009002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.009125 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.009708 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:02:34.009709 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:34.009745 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:02:34.009768 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:34.009949 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:34.009958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:34.009960 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:02:34.010006 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:02:34.010080 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260620 22:02:34.010686 25097 heartbeater.cc:347] Connected to a master server at 127.6.241.190:41651
I20260620 22:02:34.010769 25097 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:34.010917 25097 heartbeater.cc:511] Master 127.6.241.190:41651 requested a full tablet report, sending...
I20260620 22:02:34.011248 24646 ts_manager.cc:194] Registered new tserver with Master: 7c6ddfa4242042bdac12e5155a19a405 (127.6.241.131:38283)
I20260620 22:02:34.011570 24646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.131:42473
I20260620 22:02:34.020744 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:02:34.024495 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.022103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.6.241.190:41651)
0620 22:02:34.022159 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:34.022161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:34.022243 (+ 82us) server_negotiation.cc:407] Connection header received
0620 22:02:34.022278 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:34.022278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:34.022285 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:34.022328 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:34.022459 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.022970 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.023631 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.023792 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.023836 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.024325 (+ 489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:34.024326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:34.024326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:34.024329 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:34.024340 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:34.024343 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:34.024389 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:34.024430 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 22:02:34.024447 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:02:34.025835 24646 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:02:34.027376 24646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:02:34.034090 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.029381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35423 (local address 127.0.0.1:48348)
0620 22:02:34.029504 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:34.029532 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:34.029597 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:34.030398 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:34.030398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:34.030401 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:34.030449 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.030450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.031765 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.031766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.032933 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.032936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.033804 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.033804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.033880 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.033893 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:02:34.033971 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:02:34.034017 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260620 22:02:34.034103 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.029898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.6.241.130:35423)
0620 22:02:34.030133 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:34.030138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:34.030174 (+ 36us) server_negotiation.cc:407] Connection header received
0620 22:02:34.030212 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:34.030214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:34.030263 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:34.030396 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:34.030533 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.031679 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.033016 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.033721 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.033794 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.033882 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:34.033986 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 22:02:34.034023 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
I20260620 22:02:34.034778 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.029380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:38283 (local address 127.0.0.1:46650)
0620 22:02:34.029494 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:34.029532 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:34.029597 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:34.030346 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:34.030349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:34.030351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:34.030407 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.030409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.033009 (+ 2600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.033010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.033652 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.033653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.034503 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.034504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.034563 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.034567 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:02:34.034661 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 22:02:34.034706 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":80,"threads_started":1}
I20260620 22:02:34.035229 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.029594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:38925 (local address 127.0.0.1:53702)
0620 22:02:34.029892 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:34.029903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:34.029956 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:34.031267 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:34.031268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:34.031270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:34.031306 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.031307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.032653 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.032653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.033687 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.033690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.034986 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.034987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.035048 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.035051 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:34.035129 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:02:34.035167 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":61,"threads_started":1}
I20260620 22:02:34.035287 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.029694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.6.241.129:38925)
0620 22:02:34.030094 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:34.030098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:34.030107 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:02:34.030148 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:34.030149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:34.030165 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:34.030240 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:34.031400 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.032570 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.033792 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.034775 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.034866 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.034953 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:34.035149 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 22:02:34.035193 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":97,"threads_started":1}
I20260620 22:02:34.035243 24899 tablet_service.cc:1467] Processing CreateTablet for tablet a41d1bea83c24612a7da78d97cbe0bdf (DEFAULT_TABLE table=test-table [id=c6d4b1e6550e41538b50f82a2199f4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:34.035564 24899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41d1bea83c24612a7da78d97cbe0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:34.036093 24769 tablet_service.cc:1467] Processing CreateTablet for tablet a41d1bea83c24612a7da78d97cbe0bdf (DEFAULT_TABLE table=test-table [id=c6d4b1e6550e41538b50f82a2199f4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:34.036414 24769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41d1bea83c24612a7da78d97cbe0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:34.038383 25111 tablet_bootstrap.cc:492] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: Bootstrap starting.
I20260620 22:02:34.038522 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.029711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46650 (local address 127.6.241.131:38283)
0620 22:02:34.029867 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:34.029870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:34.029878 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:02:34.029903 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:34.029904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:34.029915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:34.030478 (+ 563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:34.030523 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.031686 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.033741 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.034415 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.034502 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.034578 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:34.038410 (+ 3832us) server_negotiation.cc:299] Negotiation successful
0620 22:02:34.038454 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260620 22:02:34.039027 25111 tablet_bootstrap.cc:654] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:34.039247 25111 log.cc:824] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:34.039130 25031 tablet_service.cc:1467] Processing CreateTablet for tablet a41d1bea83c24612a7da78d97cbe0bdf (DEFAULT_TABLE table=test-table [id=c6d4b1e6550e41538b50f82a2199f4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:34.039445 25031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41d1bea83c24612a7da78d97cbe0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:34.039433 25112 tablet_bootstrap.cc:492] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f: Bootstrap starting.
I20260620 22:02:34.039969 25112 tablet_bootstrap.cc:654] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:34.040073 25111 tablet_bootstrap.cc:492] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: No bootstrap required, opened a new log
I20260620 22:02:34.040132 25111 ts_tablet_manager.cc:1397] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:02:34.040271 25112 log.cc:824] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:34.041047 25112 tablet_bootstrap.cc:492] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f: No bootstrap required, opened a new log
I20260620 22:02:34.041101 25112 ts_tablet_manager.cc:1397] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:02:34.041839 25115 tablet_bootstrap.cc:492] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: Bootstrap starting.
I20260620 22:02:34.042464 25115 tablet_bootstrap.cc:654] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:34.042793 25115 log.cc:824] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:34.042732 25111 raft_consensus.cc:348] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } }
I20260620 22:02:34.042896 25111 raft_consensus.cc:374] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:34.042937 25111 raft_consensus.cc:729] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d9b825939248d3b57e06c8adcd1416, State: Initialized, Role: FOLLOWER
I20260620 22:02:34.043049 25111 consensus_queue.cc:260] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } }
I20260620 22:02:34.043305 25111 ts_tablet_manager.cc:1428] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 22:02:34.043432 24966 heartbeater.cc:503] Master 127.6.241.190:41651 was elected leader, sending a full tablet report...
I20260620 22:02:34.043489 25115 tablet_bootstrap.cc:492] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: No bootstrap required, opened a new log
I20260620 22:02:34.043555 25115 ts_tablet_manager.cc:1397] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:02:34.043628 25112 raft_consensus.cc:348] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } }
I20260620 22:02:34.043797 25112 raft_consensus.cc:374] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:34.043843 25112 raft_consensus.cc:729] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b8245d7e214f618b56dd3ec6d61e5f, State: Initialized, Role: FOLLOWER
I20260620 22:02:34.043957 25112 consensus_queue.cc:260] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } }
I20260620 22:02:34.044214 25112 ts_tablet_manager.cc:1428] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:02:34.044487 24835 heartbeater.cc:503] Master 127.6.241.190:41651 was elected leader, sending a full tablet report...
I20260620 22:02:34.045799 25115 raft_consensus.cc:348] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } }
I20260620 22:02:34.045965 25115 raft_consensus.cc:374] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:34.046010 25115 raft_consensus.cc:729] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6ddfa4242042bdac12e5155a19a405, State: Initialized, Role: FOLLOWER
I20260620 22:02:34.046111 25115 consensus_queue.cc:260] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } }
I20260620 22:02:34.046365 25115 ts_tablet_manager.cc:1428] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:02:34.046416 25097 heartbeater.cc:503] Master 127.6.241.190:41651 was elected leader, sending a full tablet report...
W20260620 22:02:34.142923 24967 tablet.cc:2369] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:02:34.208896 25119 raft_consensus.cc:482] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:34.209015 25119 raft_consensus.cc:504] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } }
I20260620 22:02:34.209299 25119 leader_election.cc:288] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3b8245d7e214f618b56dd3ec6d61e5f (127.6.241.129:38925), 57d9b825939248d3b57e06c8adcd1416 (127.6.241.130:35423)
I20260620 22:02:34.212365 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.209487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:38925 (local address 127.6.241.131:44429)
0620 22:02:34.209551 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:34.209562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:34.209611 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:34.209775 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:34.209776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:34.209778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:34.209883 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.209885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.210626 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.210627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.211518 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.211520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.212103 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.212104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.212169 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.212173 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:02:34.212234 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 22:02:34.212280 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260620 22:02:34.212412 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.209555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:44429 (local address 127.6.241.129:38925)
0620 22:02:34.209617 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:34.209619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:34.209630 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:02:34.209664 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:34.209665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:34.209686 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:34.209768 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:34.209964 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.210519 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.211597 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.212035 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.212088 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.212094 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:34.212272 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 22:02:34.212312 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:02:34.212589 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.209594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:46485 (local address 127.6.241.130:35423)
0620 22:02:34.209649 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:34.209651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:34.209838 (+ 187us) server_negotiation.cc:407] Connection header received
0620 22:02:34.209906 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:34.209907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:34.209920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:34.209967 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:34.210152 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.210719 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.211813 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.212223 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.212284 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.212290 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:34.212470 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 22:02:34.212524 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:02:34.212710 24789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41d1bea83c24612a7da78d97cbe0bdf" candidate_uuid: "7c6ddfa4242042bdac12e5155a19a405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" is_pre_election: true
I20260620 22:02:34.212787 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.209487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:35423 (local address 127.6.241.131:46485)
0620 22:02:34.209770 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:34.209777 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:34.209838 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:34.209996 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:34.209997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:34.209999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:34.210060 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.210061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.210836 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.210837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.211698 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:34.211701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.212300 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.212301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:34.212375 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.212378 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:34.212667 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 22:02:34.212706 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":102,"thread_start_us":101,"threads_started":1}
I20260620 22:02:34.212865 24789 raft_consensus.cc:2454] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6ddfa4242042bdac12e5155a19a405 in term 0.
I20260620 22:02:34.213109 24987 leader_election.cc:302] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6ddfa4242042bdac12e5155a19a405, e3b8245d7e214f618b56dd3ec6d61e5f; no voters:
I20260620 22:02:34.213156 24920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41d1bea83c24612a7da78d97cbe0bdf" candidate_uuid: "7c6ddfa4242042bdac12e5155a19a405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d9b825939248d3b57e06c8adcd1416" is_pre_election: true
I20260620 22:02:34.213260 25119 raft_consensus.cc:2781] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:02:34.213296 24920 raft_consensus.cc:2454] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6ddfa4242042bdac12e5155a19a405 in term 0.
I20260620 22:02:34.213330 25119 raft_consensus.cc:482] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:02:34.213356 25119 raft_consensus.cc:3037] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:34.214164 25119 raft_consensus.cc:504] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } }
I20260620 22:02:34.214304 25119 leader_election.cc:288] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [CANDIDATE]: Term 1 election: Requested vote from peers e3b8245d7e214f618b56dd3ec6d61e5f (127.6.241.129:38925), 57d9b825939248d3b57e06c8adcd1416 (127.6.241.130:35423)
I20260620 22:02:34.214432 24789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41d1bea83c24612a7da78d97cbe0bdf" candidate_uuid: "7c6ddfa4242042bdac12e5155a19a405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f"
I20260620 22:02:34.214452 24920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41d1bea83c24612a7da78d97cbe0bdf" candidate_uuid: "7c6ddfa4242042bdac12e5155a19a405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d9b825939248d3b57e06c8adcd1416"
I20260620 22:02:34.214510 24789 raft_consensus.cc:3037] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:34.214542 24920 raft_consensus.cc:3037] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:34.215255 24789 raft_consensus.cc:2454] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6ddfa4242042bdac12e5155a19a405 in term 1.
I20260620 22:02:34.215349 24920 raft_consensus.cc:2454] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6ddfa4242042bdac12e5155a19a405 in term 1.
I20260620 22:02:34.215418 24987 leader_election.cc:302] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6ddfa4242042bdac12e5155a19a405, e3b8245d7e214f618b56dd3ec6d61e5f; no voters:
I20260620 22:02:34.215518 25119 raft_consensus.cc:2781] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:34.215698 25119 raft_consensus.cc:686] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 1 LEADER]: Becoming Leader. State: Replica: 7c6ddfa4242042bdac12e5155a19a405, State: Running, Role: LEADER
I20260620 22:02:34.215802 25119 consensus_queue.cc:237] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } }
I20260620 22:02:34.216601 24646 catalog_manager.cc:5495] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c6ddfa4242042bdac12e5155a19a405 (127.6.241.131). New cstate: current_term: 1 leader_uuid: "7c6ddfa4242042bdac12e5155a19a405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6ddfa4242042bdac12e5155a19a405" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 38283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 } health_report { overall_health: UNKNOWN } } }
W20260620 22:02:34.240556 24836 tablet.cc:2369] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:02:34.255525 25098 tablet.cc:2369] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:02:34.259531 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.257420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.6.241.131:38283)
0620 22:02:34.257483 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:34.257485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:34.257493 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:02:34.257525 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:34.257525 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:34.257541 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:34.257612 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:02:34.257722 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.258182 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.258901 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:34.259079 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:34.259131 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:34.259255 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:02:34.259354 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:02:34.259413 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:34.259446 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 22:02:34.259461 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:02:34.261265 24789 raft_consensus.cc:1261] T a41d1bea83c24612a7da78d97cbe0bdf P e3b8245d7e214f618b56dd3ec6d61e5f [term 1 FOLLOWER]: Refusing update from remote peer 7c6ddfa4242042bdac12e5155a19a405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:34.261281 24920 raft_consensus.cc:1261] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [term 1 FOLLOWER]: Refusing update from remote peer 7c6ddfa4242042bdac12e5155a19a405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:34.261632 25121 consensus_queue.cc:1035] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b8245d7e214f618b56dd3ec6d61e5f" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 38925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:34.261765 25119 consensus_queue.cc:1035] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d9b825939248d3b57e06c8adcd1416" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:34.264526 25125 mvcc.cc:205] Tried to move back new op lower bound from 7299043140651364352 to 7299043140468674560. Current Snapshot: MvccSnapshot[applied={T|T < 7299043140651364352 or (T in {7299043140651364352})}]
I20260620 22:02:34.266522 24645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:34.266638 24645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:34.267443 24645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 788307795c24466197dc71c36cdb4154: Sending DeleteTablet for 3 replicas of tablet a41d1bea83c24612a7da78d97cbe0bdf
I20260620 22:02:34.267807 24899 tablet_service.cc:1514] Processing DeleteTablet for tablet a41d1bea83c24612a7da78d97cbe0bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:34 UTC) from {username='slave'} at 127.0.0.1:48348
I20260620 22:02:34.268011 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 24707
I20260620 22:02:34.268004 24769 tablet_service.cc:1514] Processing DeleteTablet for tablet a41d1bea83c24612a7da78d97cbe0bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:34 UTC) from {username='slave'} at 127.0.0.1:53702
I20260620 22:02:34.268661 25031 tablet_service.cc:1514] Processing DeleteTablet for tablet a41d1bea83c24612a7da78d97cbe0bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:34 UTC) from {username='slave'} at 127.0.0.1:46650
I20260620 22:02:34.268852 25135 tablet_replica.cc:330] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: stopping tablet replica
I20260620 22:02:34.268957 25135 raft_consensus.cc:2229] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:02:34.269272 25134 tablet_replica.cc:330] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: stopping tablet replica
I20260620 22:02:34.269358 25134 raft_consensus.cc:2229] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:02:34.269469 25134 raft_consensus.cc:2258] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:34.269814 25134 ts_tablet_manager.cc:1905] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:34.271247 25135 raft_consensus.cc:2258] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:34.271304 25134 ts_tablet_manager.cc:1918] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:34.271386 25134 log.cc:1196] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/a41d1bea83c24612a7da78d97cbe0bdf
I20260620 22:02:34.271656 25134 ts_tablet_manager.cc:1939] T a41d1bea83c24612a7da78d97cbe0bdf P 57d9b825939248d3b57e06c8adcd1416: Deleting consensus metadata
I20260620 22:02:34.272044 24633 catalog_manager.cc:4841] TS 57d9b825939248d3b57e06c8adcd1416 (127.6.241.130:35423): tablet a41d1bea83c24612a7da78d97cbe0bdf (table test-table [id=c6d4b1e6550e41538b50f82a2199f4ae]) successfully deleted
W20260620 22:02:34.272624 24634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:38925 (error 108)
I20260620 22:02:34.273056 25135 ts_tablet_manager.cc:1905] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:34.273648 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 24838
I20260620 22:02:34.274825 25135 ts_tablet_manager.cc:1918] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:34.274906 25135 log.cc:1196] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/a41d1bea83c24612a7da78d97cbe0bdf
I20260620 22:02:34.275156 25135 ts_tablet_manager.cc:1939] T a41d1bea83c24612a7da78d97cbe0bdf P 7c6ddfa4242042bdac12e5155a19a405: Deleting consensus metadata
I20260620 22:02:34.276923 24634 catalog_manager.cc:4841] TS 7c6ddfa4242042bdac12e5155a19a405 (127.6.241.131:38283): tablet a41d1bea83c24612a7da78d97cbe0bdf (table test-table [id=c6d4b1e6550e41538b50f82a2199f4ae]) successfully deleted
I20260620 22:02:34.277045 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:34.276874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:38925 (local address 127.0.0.1:53714)
0620 22:02:34.276931 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:34.276977 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:38925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 22:02:34.277165 24634 catalog_manager.cc:4584] TS e3b8245d7e214f618b56dd3ec6d61e5f (127.6.241.129:38925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a41d1bea83c24612a7da78d97cbe0bdf: Network error: Client connection negotiation failed: client connection to 127.6.241.129:38925: connect: Connection refused (error 111)
I20260620 22:02:34.281251 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 24969
I20260620 22:02:34.285729 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 24616
2026-06-20T22:02:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (869 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:34 dist-test-slave-gl2g krb5kdc[25139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:34 dist-test-slave-gl2g krb5kdc[25139](info): set up 2 sockets
Jun 20 22:02:34 dist-test-slave-gl2g krb5kdc[25139](info): commencing operation
krb5kdc: starting...
W20260620 22:02:35.310286 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:35 dist-test-slave-gl2g krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:02:35.416877 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:46355
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:46355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:35.493180 25152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:35.493358 25152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:35.493386 25152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:35.494515 25152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:35.494554 25152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:35.494568 25152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:35.494606 25152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:35.494655 25152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:35.496090 25152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:46355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:46355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:35.496320 25152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:35.496558 25152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:35.501183 25157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:35.501288 25152 server_base.cc:1034] running on GCE node
W20260620 22:02:35.501194 25158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:35.501209 25160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:35.501603 25152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:35.501883 25152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:35.503041 25152 hybrid_clock.cc:648] HybridClock initialized: now 1781992955503036 us; error 36 us; skew 500 ppm
Jun 20 22:02:35 dist-test-slave-gl2g krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992955, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:35.505368 25152 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:02:35.505733 25152 webserver.cc:460] Webserver started at http://127.6.241.190:37049/ using document root <none> and password file <none>
I20260620 22:02:35.505911 25152 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:35.505950 25152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:35.506033 25152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:35.507011 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "b04ec3afe65b40349de468e17bd3c4f8"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "dcf3170d860eb11d8bbc207329469ef8"
server_key_iv: "662906be3d6c7704264985226b10393c"
server_key_version: "encryptionkey@0"
I20260620 22:02:35.507311 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "b04ec3afe65b40349de468e17bd3c4f8"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "dcf3170d860eb11d8bbc207329469ef8"
server_key_iv: "662906be3d6c7704264985226b10393c"
server_key_version: "encryptionkey@0"
I20260620 22:02:35.508467 25152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:02:35.509078 25167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:35.509239 25152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:35.509322 25152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "b04ec3afe65b40349de468e17bd3c4f8"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "dcf3170d860eb11d8bbc207329469ef8"
server_key_iv: "662906be3d6c7704264985226b10393c"
server_key_version: "encryptionkey@0"
I20260620 22:02:35.509397 25152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:35.523726 25152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:35.525848 25152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:35.526005 25152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:35.529652 25152 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:46355
I20260620 22:02:35.529704 25219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:46355 every 8 connection(s)
I20260620 22:02:35.530030 25152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:35.530916 25220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:35.531018 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 25152
I20260620 22:02:35.531158 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:35.531343 7110 external_mini_cluster.cc:1428] Setting key f6d93d27ac249b37a1960a59036cb4d2
I20260620 22:02:35.534096 25220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8: Bootstrap starting.
I20260620 22:02:35.534788 25220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:35.535063 25220 log.cc:824] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:35.535712 25220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8: No bootstrap required, opened a new log
I20260620 22:02:35.537820 25220 raft_consensus.cc:348] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ec3afe65b40349de468e17bd3c4f8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 46355 } }
I20260620 22:02:35.537985 25220 raft_consensus.cc:374] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:35.538017 25220 raft_consensus.cc:729] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04ec3afe65b40349de468e17bd3c4f8, State: Initialized, Role: FOLLOWER
Jun 20 22:02:35 dist-test-slave-gl2g krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:35.538170 25220 consensus_queue.cc:260] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ec3afe65b40349de468e17bd3c4f8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 46355 } }
I20260620 22:02:35.538252 25220 raft_consensus.cc:388] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:35.538301 25220 raft_consensus.cc:482] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:35.538358 25220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:35.539003 25220 raft_consensus.cc:504] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ec3afe65b40349de468e17bd3c4f8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 46355 } }
I20260620 22:02:35.539110 25220 leader_election.cc:302] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b04ec3afe65b40349de468e17bd3c4f8; no voters:
I20260620 22:02:35.539336 25220 leader_election.cc:288] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:35.539368 25225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:35.539601 25220 sys_catalog.cc:564] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:35.539718 25225 raft_consensus.cc:686] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [term 1 LEADER]: Becoming Leader. State: Replica: b04ec3afe65b40349de468e17bd3c4f8, State: Running, Role: LEADER
I20260620 22:02:35.539819 25225 consensus_queue.cc:237] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ec3afe65b40349de468e17bd3c4f8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 46355 } }
I20260620 22:02:35.540119 25225 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04ec3afe65b40349de468e17bd3c4f8. Latest consensus state: current_term: 1 leader_uuid: "b04ec3afe65b40349de468e17bd3c4f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ec3afe65b40349de468e17bd3c4f8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 46355 } } }
I20260620 22:02:35.540127 25226 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b04ec3afe65b40349de468e17bd3c4f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ec3afe65b40349de468e17bd3c4f8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 46355 } } }
I20260620 22:02:35.540211 25225 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:35.540230 25226 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:35.540395 25234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:35.540865 25234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:35.542351 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:35.531844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.6.241.190:46355)
0620 22:02:35.532056 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:35.532065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:35.532096 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:02:35.532548 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:35.532556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:35.532615 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:35.532767 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:35.533347 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.534707 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.535535 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.535988 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.536115 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:35.538638 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:35.538642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:35.538643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:35.538654 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:35.540579 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:35.541181 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:35.541185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:35.541185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:35.541278 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:35.541523 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:35.541524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:35.541524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:35.541771 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:35.541939 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:35.542048 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 22:02:35.542181 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260620 22:02:35.543241 25234 catalog_manager.cc:1347] Generated new cluster ID: 8bd60be7cbeb4a53becb7379e146531b
I20260620 22:02:35.543298 25234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:35.564776 25234 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:35.565507 25234 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:35.572726 25234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8: Generated new TSK 0
I20260620 22:02:35.572918 25234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 22:02:35.605310 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:46355
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:35.682859 25246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:35.683044 25246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:35.683068 25246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:35.684237 25246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:35.684278 25246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:35.684304 25246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:02:35.685737 25246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:35.685933 25246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:35.686196 25246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:35.691067 25251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:35.691068 25254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:35.691093 25252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:35.691470 25246 server_base.cc:1034] running on GCE node
I20260620 22:02:35.691704 25246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:35.691936 25246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:35.693092 25246 hybrid_clock.cc:648] HybridClock initialized: now 1781992955693089 us; error 36 us; skew 500 ppm
Jun 20 22:02:35 dist-test-slave-gl2g krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992955, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:35.695533 25246 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:02:35.695966 25246 webserver.cc:460] Webserver started at http://127.6.241.129:39145/ using document root <none> and password file <none>
I20260620 22:02:35.696178 25246 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:35.696245 25246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:35.696398 25246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:35.697393 25246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "a82029e2aa5643289cde21aed06365ba"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "c4bb9ba37ed107913e7c55cfc4a6637c"
server_key_iv: "09ad9c63ee6acb392f4350c9f4774994"
server_key_version: "encryptionkey@0"
I20260620 22:02:35.697767 25246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "a82029e2aa5643289cde21aed06365ba"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "c4bb9ba37ed107913e7c55cfc4a6637c"
server_key_iv: "09ad9c63ee6acb392f4350c9f4774994"
server_key_version: "encryptionkey@0"
I20260620 22:02:35.699082 25246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:35.699769 25261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:35.699939 25246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:35.700033 25246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "a82029e2aa5643289cde21aed06365ba"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "c4bb9ba37ed107913e7c55cfc4a6637c"
server_key_iv: "09ad9c63ee6acb392f4350c9f4774994"
server_key_version: "encryptionkey@0"
I20260620 22:02:35.700110 25246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:35.716485 25246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:35.718310 25246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:35.718466 25246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:35.718729 25246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:35.719038 25246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:35.719094 25246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:35.719132 25246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:35.719161 25246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:35.724465 25246 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:42743
I20260620 22:02:35.724539 25374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:42743 every 8 connection(s)
I20260620 22:02:35.724831 25246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:02:35.729769 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 25246
I20260620 22:02:35.729873 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:02:35.730049 7110 external_mini_cluster.cc:1428] Setting key ee91b18954fb2dbb14567fe5ee8c4956
Jun 20 22:02:35 dist-test-slave-gl2g krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992955, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:35.734513 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:35.725556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:46355 (local address 127.6.241.129:58973)
0620 22:02:35.725717 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:35.725763 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:35.726206 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:35.726395 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:35.726398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:35.726951 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:35.727127 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:35.727132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.727778 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.727779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:35.728953 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:35.728956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.729388 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.729389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:35.729487 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:35.729977 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:35.729986 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:35.731546 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:35.733175 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:35.733180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:35.733182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:35.733430 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:35.733647 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:35.733647 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:35.733648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:35.733763 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:35.734061 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:35.734065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:35.734165 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 22:02:35.734242 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:02:35.734355 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260620 22:02:35.734525 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:35.725575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:58973 (local address 127.6.241.190:46355)
0620 22:02:35.725643 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:35.725645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:35.725854 (+ 209us) server_negotiation.cc:407] Connection header received
0620 22:02:35.726284 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:35.726285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:35.726293 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:35.726347 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:35.727210 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.727692 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.729030 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.729318 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.729377 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:35.731647 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:35.731649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:35.731650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:35.731657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:35.733087 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:35.733514 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:35.733515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:35.733515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:35.733576 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:35.733848 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:35.733849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:35.733850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:35.733997 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:35.734067 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:35.734295 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 22:02:35.734389 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:02:35.735057 25375 heartbeater.cc:347] Connected to a master server at 127.6.241.190:46355
I20260620 22:02:35.735162 25375 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:35.735342 25375 heartbeater.cc:511] Master 127.6.241.190:46355 requested a full tablet report, sending...
I20260620 22:02:35.735814 25184 ts_manager.cc:194] Registered new tserver with Master: a82029e2aa5643289cde21aed06365ba (127.6.241.129:42743)
I20260620 22:02:35.736418 25184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:58973
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 22:02:35.753424 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:46355
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:35.833146 25380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:35.833335 25380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:35.833357 25380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:35.834442 25380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:35.834476 25380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:35.834502 25380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:02:35.835964 25380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:35.836143 25380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:35.836306 25380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:35.841465 25385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:35.841528 25380 server_base.cc:1034] running on GCE node
W20260620 22:02:35.841504 25386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:35.841481 25388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:35.841754 25380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:35.841986 25380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:35.843133 25380 hybrid_clock.cc:648] HybridClock initialized: now 1781992955843119 us; error 42 us; skew 500 ppm
Jun 20 22:02:35 dist-test-slave-gl2g krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992955, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:35.845546 25380 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:02:35.845994 25380 webserver.cc:460] Webserver started at http://127.6.241.130:42829/ using document root <none> and password file <none>
I20260620 22:02:35.846220 25380 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:35.846261 25380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:35.846405 25380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:35.847519 25380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "38625a9761b944abbb392f90e6abe710"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "269b7a66ad1ece4e8eb2a360c3ba6d95"
server_key_iv: "bf2d25c70d653142ed2b85c3d270238e"
server_key_version: "encryptionkey@0"
I20260620 22:02:35.847858 25380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "38625a9761b944abbb392f90e6abe710"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "269b7a66ad1ece4e8eb2a360c3ba6d95"
server_key_iv: "bf2d25c70d653142ed2b85c3d270238e"
server_key_version: "encryptionkey@0"
I20260620 22:02:35.849098 25380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:02:35.849787 25395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:35.849982 25380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:02:35.850080 25380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "38625a9761b944abbb392f90e6abe710"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "269b7a66ad1ece4e8eb2a360c3ba6d95"
server_key_iv: "bf2d25c70d653142ed2b85c3d270238e"
server_key_version: "encryptionkey@0"
I20260620 22:02:35.850162 25380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:35.876634 25380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:35.878369 25380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:35.878530 25380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:35.878767 25380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:35.879057 25380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:35.879113 25380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:35.879153 25380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:35.879180 25380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:35.884852 25380 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:33213
I20260620 22:02:35.885032 25508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:33213 every 8 connection(s)
I20260620 22:02:35.885242 25380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:02:35.887810 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 25380
I20260620 22:02:35.887923 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:02:35.888104 7110 external_mini_cluster.cc:1428] Setting key 0cb1504c8734e464a498894ae99047bf
Jun 20 22:02:35 dist-test-slave-gl2g krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992955, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:35.894780 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:35.885935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:38389 (local address 127.6.241.190:46355)
0620 22:02:35.885997 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:35.886000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:35.886214 (+ 214us) server_negotiation.cc:407] Connection header received
0620 22:02:35.886696 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:35.886697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:35.886714 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:35.886754 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:35.887648 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.888295 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.889295 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.889512 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.889604 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:35.891981 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:35.891983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:35.891983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:35.891991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:35.893381 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:35.893804 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:35.893806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:35.893806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:35.893865 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:35.894124 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:35.894125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:35.894126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:35.894238 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:35.894313 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:35.894583 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 22:02:35.894690 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:02:35.894963 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:35.885926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:46355 (local address 127.6.241.130:38389)
0620 22:02:35.886092 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:35.886131 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:35.886586 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:35.886818 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:35.886821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:35.887353 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:35.887560 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:35.887565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.888399 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.888400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:35.889193 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:35.889196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:35.889614 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:35.889615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:35.889741 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:35.890261 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:35.890269 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:35.891871 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:35.893483 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:35.893490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:35.893492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:35.893713 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:35.893939 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:35.893940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:35.893941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:35.894054 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:35.894305 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:35.894312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:35.894453 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 22:02:35.894562 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 22:02:35.894737 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260620 22:02:35.895423 25509 heartbeater.cc:347] Connected to a master server at 127.6.241.190:46355
I20260620 22:02:35.895514 25509 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:35.895704 25509 heartbeater.cc:511] Master 127.6.241.190:46355 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
I20260620 22:02:35.896032 25184 ts_manager.cc:194] Registered new tserver with Master: 38625a9761b944abbb392f90e6abe710 (127.6.241.130:33213)
I20260620 22:02:35.896474 25184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:38389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 22:02:35.911064 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:46355
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:35.987838 25514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:35.988014 25514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:35.988036 25514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:35.989068 25514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:35.989100 25514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:35.989125 25514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:02:35.990393 25514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:35.990553 25514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:35.990717 25514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:35.995283 25520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:35.995296 25522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:35.995301 25519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:35.995477 25514 server_base.cc:1034] running on GCE node
I20260620 22:02:35.995668 25514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:35.995918 25514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:35.997076 25514 hybrid_clock.cc:648] HybridClock initialized: now 1781992955997070 us; error 35 us; skew 500 ppm
Jun 20 22:02:35 dist-test-slave-gl2g krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992955, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:35.999653 25514 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:02:36.000005 25514 webserver.cc:460] Webserver started at http://127.6.241.131:35993/ using document root <none> and password file <none>
I20260620 22:02:36.000165 25514 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:36.000204 25514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:36.000293 25514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:36.001156 25514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "ad6ea6cb53e94c98b87933d07c0b2353"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "74057ea62833f326e56124c9d8bfc40d"
server_key_iv: "f26116c74948d31f0a151a37e968a597"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.001442 25514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "ad6ea6cb53e94c98b87933d07c0b2353"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "74057ea62833f326e56124c9d8bfc40d"
server_key_iv: "f26116c74948d31f0a151a37e968a597"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.002583 25514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:02:36.003422 25529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.003593 25514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:02:36.003703 25514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "ad6ea6cb53e94c98b87933d07c0b2353"
format_stamp: "Formatted at 2026-06-20 22:02:35 on dist-test-slave-gl2g"
server_key: "74057ea62833f326e56124c9d8bfc40d"
server_key_iv: "f26116c74948d31f0a151a37e968a597"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.003796 25514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:36.027807 25514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:36.029481 25514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:36.029655 25514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:36.029882 25514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:36.030201 25514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:36.030264 25514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.030301 25514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:36.030329 25514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.035837 25514 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:40773
I20260620 22:02:36.035892 25642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:40773 every 8 connection(s)
I20260620 22:02:36.036211 25514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:02:36 dist-test-slave-gl2g krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992955, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:36.045100 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.036916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:55611 (local address 127.6.241.190:46355)
0620 22:02:36.036957 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.036959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.037155 (+ 196us) server_negotiation.cc:407] Connection header received
0620 22:02:36.037597 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.037598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.037607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.037652 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:36.038547 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.039033 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.040058 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.040289 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.040392 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.042488 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:36.042489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:36.042490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:36.042497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:36.043734 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:36.044158 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:36.044159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:36.044159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:36.044221 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:36.044472 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:36.044473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:36.044474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:36.044589 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:36.044664 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.044922 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.045012 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:02:36.045258 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.036873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:46355 (local address 127.6.241.131:55611)
0620 22:02:36.037022 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.037060 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.037520 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.037727 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.037731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.038292 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:36.038475 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.038479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.039107 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.039108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.039987 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.039989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.040428 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.040429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.040553 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.041003 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:36.041010 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:36.042392 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:36.043826 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:36.043832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:36.043834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:36.044066 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:36.044288 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:36.044289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:36.044289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:36.044403 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:36.044655 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:36.044660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:36.044790 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.044895 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.045040 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260620 22:02:36.045526 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 25514
I20260620 22:02:36.045603 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:02:36.045697 25643 heartbeater.cc:347] Connected to a master server at 127.6.241.190:46355
I20260620 22:02:36.045787 25643 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:36.045797 7110 external_mini_cluster.cc:1428] Setting key 5e2f548c0219d90ccf4b0ee3f295ee27
I20260620 22:02:36.045966 25643 heartbeater.cc:511] Master 127.6.241.190:46355 requested a full tablet report, sending...
I20260620 22:02:36.046339 25184 ts_manager.cc:194] Registered new tserver with Master: ad6ea6cb53e94c98b87933d07c0b2353 (127.6.241.131:40773)
I20260620 22:02:36.046591 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:02:36.046810 25184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:55611
I20260620 22:02:36.053432 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.047826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.6.241.190:46355)
0620 22:02:36.047857 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.047859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.047982 (+ 123us) server_negotiation.cc:407] Connection header received
0620 22:02:36.048035 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.048036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.048044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.048086 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:36.048565 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.048995 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.049644 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.049812 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.049859 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.051074 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:36.051075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:36.051075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:36.051081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:36.052400 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:36.052794 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:36.052794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:36.052794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:36.052842 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:36.053016 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:36.053016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:36.053016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:36.053123 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:36.053193 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.053332 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.053377 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:02:36.054963 25184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:02:36.056499 25184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:02:36.062929 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.058583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:40773 (local address 127.0.0.1:38784)
0620 22:02:36.058899 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.058909 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.058946 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.059467 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.059468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.059470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.059517 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.059519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.060776 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.060777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.061818 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.061820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.062662 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.062662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.062711 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.062713 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.062793 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.062846 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":17,"thread_start_us":69,"threads_started":1}
I20260620 22:02:36.062928 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.058706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.6.241.131:40773)
0620 22:02:36.059321 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.059325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.059335 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:02:36.059371 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.059373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.059387 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.059472 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.059581 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.060671 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.061916 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.062564 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.062652 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.062705 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.062806 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.062846 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":107,"threads_started":1}
I20260620 22:02:36.063040 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.058906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.6.241.129:42743)
0620 22:02:36.059060 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.059065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.059076 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:02:36.059118 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.059120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.059137 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.059240 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.059521 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.060374 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.061766 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.062442 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.062532 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.062592 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.062904 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.062951 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260620 22:02:36.063148 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.058583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42743 (local address 127.0.0.1:38158)
0620 22:02:36.058751 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.058788 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.058848 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.059354 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.059357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.059359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.059434 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.059436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.060478 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.060479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.061674 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.061676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.062580 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.062581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.062699 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.062713 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.062783 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.062846 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260620 22:02:36.063328 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.058782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:33213 (local address 127.0.0.1:48928)
0620 22:02:36.059559 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.059570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.059623 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.059793 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.059794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.059795 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.059830 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.059831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.061011 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.061012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.062070 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.062073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.063087 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.063088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.063162 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.063166 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.063210 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.063248 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260620 22:02:36.063548 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.059099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.6.241.130:33213)
0620 22:02:36.059303 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.059307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.059623 (+ 316us) server_negotiation.cc:407] Connection header received
0620 22:02:36.059689 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.059691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.059707 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.059782 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.059892 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.060903 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.062180 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.062919 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.062986 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.063046 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.063421 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.063477 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260620 22:02:36.064057 25309 tablet_service.cc:1467] Processing CreateTablet for tablet b2fe7e444e0f44858f52cdce36802252 (DEFAULT_TABLE table=test-table [id=9e925792af884fe8bf8e781e8d550d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:36.064143 25577 tablet_service.cc:1467] Processing CreateTablet for tablet b2fe7e444e0f44858f52cdce36802252 (DEFAULT_TABLE table=test-table [id=9e925792af884fe8bf8e781e8d550d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:36.064139 25443 tablet_service.cc:1467] Processing CreateTablet for tablet b2fe7e444e0f44858f52cdce36802252 (DEFAULT_TABLE table=test-table [id=9e925792af884fe8bf8e781e8d550d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:36.064347 25309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fe7e444e0f44858f52cdce36802252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:36.064451 25443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fe7e444e0f44858f52cdce36802252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:36.064451 25577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fe7e444e0f44858f52cdce36802252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:36.067085 25657 tablet_bootstrap.cc:492] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba: Bootstrap starting.
I20260620 22:02:36.067619 25658 tablet_bootstrap.cc:492] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: Bootstrap starting.
I20260620 22:02:36.067804 25657 tablet_bootstrap.cc:654] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:36.067828 25659 tablet_bootstrap.cc:492] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: Bootstrap starting.
I20260620 22:02:36.068411 25657 log.cc:824] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:36.068504 25659 tablet_bootstrap.cc:654] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:36.069054 25659 log.cc:824] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:36.069197 25657 tablet_bootstrap.cc:492] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba: No bootstrap required, opened a new log
I20260620 22:02:36.069268 25657 ts_tablet_manager.cc:1397] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:02:36.069717 25659 tablet_bootstrap.cc:492] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: No bootstrap required, opened a new log
I20260620 22:02:36.069799 25659 ts_tablet_manager.cc:1397] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:02:36.069968 25658 tablet_bootstrap.cc:654] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:36.070485 25658 log.cc:824] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:36.071254 25658 tablet_bootstrap.cc:492] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: No bootstrap required, opened a new log
I20260620 22:02:36.071321 25658 ts_tablet_manager.cc:1397] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 22:02:36.071344 25657 raft_consensus.cc:348] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } }
I20260620 22:02:36.071523 25657 raft_consensus.cc:374] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:36.071560 25657 raft_consensus.cc:729] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82029e2aa5643289cde21aed06365ba, State: Initialized, Role: FOLLOWER
I20260620 22:02:36.071446 25659 raft_consensus.cc:348] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } }
I20260620 22:02:36.071599 25659 raft_consensus.cc:374] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:36.071641 25659 raft_consensus.cc:729] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6ea6cb53e94c98b87933d07c0b2353, State: Initialized, Role: FOLLOWER
I20260620 22:02:36.071691 25657 consensus_queue.cc:260] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } }
I20260620 22:02:36.071771 25659 consensus_queue.cc:260] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } }
I20260620 22:02:36.071978 25657 ts_tablet_manager.cc:1428] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:02:36.072069 25659 ts_tablet_manager.cc:1428] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:02:36.072095 25375 heartbeater.cc:503] Master 127.6.241.190:46355 was elected leader, sending a full tablet report...
I20260620 22:02:36.072117 25643 heartbeater.cc:503] Master 127.6.241.190:46355 was elected leader, sending a full tablet report...
I20260620 22:02:36.073324 25658 raft_consensus.cc:348] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } }
I20260620 22:02:36.073506 25658 raft_consensus.cc:374] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:36.073532 25658 raft_consensus.cc:729] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38625a9761b944abbb392f90e6abe710, State: Initialized, Role: FOLLOWER
I20260620 22:02:36.073630 25658 consensus_queue.cc:260] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } }
I20260620 22:02:36.073900 25658 ts_tablet_manager.cc:1428] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:02:36.073920 25509 heartbeater.cc:503] Master 127.6.241.190:46355 was elected leader, sending a full tablet report...
W20260620 22:02:36.135788 25510 tablet.cc:2369] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:02:36.225732 25376 tablet.cc:2369] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:02:36.254159 25664 raft_consensus.cc:482] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:36.254274 25664 raft_consensus.cc:504] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } }
I20260620 22:02:36.254628 25664 leader_election.cc:288] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a82029e2aa5643289cde21aed06365ba (127.6.241.129:42743), 38625a9761b944abbb392f90e6abe710 (127.6.241.130:33213)
I20260620 22:02:36.257473 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.254839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42743 (local address 127.6.241.131:36121)
0620 22:02:36.254998 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.255012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.255071 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.255234 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.255235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.255236 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.255310 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.255311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.255930 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.255932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.256647 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.256649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.257208 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.257209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.257291 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.257293 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.257361 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.257402 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260620 22:02:36.257473 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.255016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:36121 (local address 127.6.241.129:42743)
0620 22:02:36.255063 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.255065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.255075 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:02:36.255141 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.255142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.255152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.255227 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.255391 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.255839 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.256741 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.257095 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.257210 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.257218 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.257368 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.257409 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:02:36.257614 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.255021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:47967 (local address 127.6.241.130:33213)
0620 22:02:36.255067 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.255069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.255204 (+ 135us) server_negotiation.cc:407] Connection header received
0620 22:02:36.255275 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.255277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.255288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.255358 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.255510 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.255952 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.256878 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.257268 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.257357 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.257364 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.257515 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.257551 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:02:36.257625 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.254960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:33213 (local address 127.6.241.131:47967)
0620 22:02:36.255124 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.255132 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.255211 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.255362 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.255363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.255365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.255432 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.255434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.256038 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.256038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.256782 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.256783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.257357 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.257358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.257427 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.257431 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.257509 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.257545 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260620 22:02:36.257822 25329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fe7e444e0f44858f52cdce36802252" candidate_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82029e2aa5643289cde21aed06365ba" is_pre_election: true
I20260620 22:02:36.257988 25329 raft_consensus.cc:2454] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6ea6cb53e94c98b87933d07c0b2353 in term 0.
I20260620 22:02:36.258253 25533 leader_election.cc:302] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a82029e2aa5643289cde21aed06365ba, ad6ea6cb53e94c98b87933d07c0b2353; no voters:
I20260620 22:02:36.258421 25664 raft_consensus.cc:2781] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:02:36.258490 25664 raft_consensus.cc:482] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:02:36.258527 25664 raft_consensus.cc:3037] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:36.258484 25463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fe7e444e0f44858f52cdce36802252" candidate_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38625a9761b944abbb392f90e6abe710" is_pre_election: true
I20260620 22:02:36.258670 25463 raft_consensus.cc:2454] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6ea6cb53e94c98b87933d07c0b2353 in term 0.
I20260620 22:02:36.259414 25664 raft_consensus.cc:504] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } }
I20260620 22:02:36.259554 25664 leader_election.cc:288] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [CANDIDATE]: Term 1 election: Requested vote from peers a82029e2aa5643289cde21aed06365ba (127.6.241.129:42743), 38625a9761b944abbb392f90e6abe710 (127.6.241.130:33213)
I20260620 22:02:36.259763 25463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fe7e444e0f44858f52cdce36802252" candidate_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38625a9761b944abbb392f90e6abe710"
I20260620 22:02:36.259739 25329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fe7e444e0f44858f52cdce36802252" candidate_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82029e2aa5643289cde21aed06365ba"
I20260620 22:02:36.259835 25463 raft_consensus.cc:3037] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:36.259835 25329 raft_consensus.cc:3037] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:36.260705 25463 raft_consensus.cc:2454] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6ea6cb53e94c98b87933d07c0b2353 in term 1.
I20260620 22:02:36.260705 25329 raft_consensus.cc:2454] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6ea6cb53e94c98b87933d07c0b2353 in term 1.
I20260620 22:02:36.260875 25533 leader_election.cc:302] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38625a9761b944abbb392f90e6abe710, ad6ea6cb53e94c98b87933d07c0b2353; no voters:
I20260620 22:02:36.260972 25664 raft_consensus.cc:2781] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:36.261137 25664 raft_consensus.cc:686] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 1 LEADER]: Becoming Leader. State: Replica: ad6ea6cb53e94c98b87933d07c0b2353, State: Running, Role: LEADER
I20260620 22:02:36.261253 25664 consensus_queue.cc:237] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } }
I20260620 22:02:36.262137 25183 catalog_manager.cc:5495] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad6ea6cb53e94c98b87933d07c0b2353 (127.6.241.131). New cstate: current_term: 1 leader_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ea6cb53e94c98b87933d07c0b2353" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 40773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 } health_report { overall_health: UNKNOWN } } }
W20260620 22:02:36.286768 25644 tablet.cc:2369] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:02:36.287348 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.285097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.6.241.131:40773)
0620 22:02:36.285166 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.285168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.285176 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:02:36.285206 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.285206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.285221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.285290 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:02:36.285378 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.285836 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.286555 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.286848 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.286918 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.287064 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:02:36.287149 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:02:36.287220 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.287275 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.287285 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:02:36.289029 25463 raft_consensus.cc:1261] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [term 1 FOLLOWER]: Refusing update from remote peer ad6ea6cb53e94c98b87933d07c0b2353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:36.289065 25329 raft_consensus.cc:1261] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [term 1 FOLLOWER]: Refusing update from remote peer ad6ea6cb53e94c98b87933d07c0b2353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:36.289402 25664 consensus_queue.cc:1035] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38625a9761b944abbb392f90e6abe710" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:36.289531 25667 consensus_queue.cc:1035] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82029e2aa5643289cde21aed06365ba" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:36.291915 25670 mvcc.cc:205] Tried to move back new op lower bound from 7299043148957327360 to 7299043148846833664. Current Snapshot: MvccSnapshot[applied={T|T < 7299043148957327360}]
I20260620 22:02:36.293005 25672 mvcc.cc:205] Tried to move back new op lower bound from 7299043148957327360 to 7299043148846833664. Current Snapshot: MvccSnapshot[applied={T|T < 7299043148957327360 or (T in {7299043148957327360})}]
I20260620 22:02:36.295192 25184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:36.295267 25184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:36.296124 25184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b04ec3afe65b40349de468e17bd3c4f8: Sending DeleteTablet for 3 replicas of tablet b2fe7e444e0f44858f52cdce36802252
I20260620 22:02:36.296471 25577 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fe7e444e0f44858f52cdce36802252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:36 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:38784
I20260620 22:02:36.296511 25309 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fe7e444e0f44858f52cdce36802252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:36 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:38158
I20260620 22:02:36.296659 25443 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fe7e444e0f44858f52cdce36802252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:36 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:48928
I20260620 22:02:36.296715 25681 tablet_replica.cc:330] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba: stopping tablet replica
I20260620 22:02:36.296800 25681 raft_consensus.cc:2229] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:02:36.296900 25681 raft_consensus.cc:2258] T b2fe7e444e0f44858f52cdce36802252 P a82029e2aa5643289cde21aed06365ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:36.296991 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 25246
I20260620 22:02:36.298812 25682 tablet_replica.cc:330] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: stopping tablet replica
I20260620 22:02:36.298929 25680 tablet_replica.cc:330] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: stopping tablet replica
I20260620 22:02:36.299057 25682 raft_consensus.cc:2229] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:02:36.299154 25680 raft_consensus.cc:2229] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:02:36.299250 25680 raft_consensus.cc:2258] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:36.299409 25682 raft_consensus.cc:2258] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:36.299753 25680 ts_tablet_manager.cc:1905] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:36.299860 25682 ts_tablet_manager.cc:1905] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:36.301632 25682 ts_tablet_manager.cc:1918] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:36.301707 25682 log.cc:1196] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/b2fe7e444e0f44858f52cdce36802252
I20260620 22:02:36.301903 25680 ts_tablet_manager.cc:1918] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:36.301990 25682 ts_tablet_manager.cc:1939] T b2fe7e444e0f44858f52cdce36802252 P 38625a9761b944abbb392f90e6abe710: Deleting consensus metadata
I20260620 22:02:36.302022 25680 log.cc:1196] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/b2fe7e444e0f44858f52cdce36802252
I20260620 22:02:36.302388 25680 ts_tablet_manager.cc:1939] T b2fe7e444e0f44858f52cdce36802252 P ad6ea6cb53e94c98b87933d07c0b2353: Deleting consensus metadata
I20260620 22:02:36.302383 25171 catalog_manager.cc:4841] TS 38625a9761b944abbb392f90e6abe710 (127.6.241.130:33213): tablet b2fe7e444e0f44858f52cdce36802252 (table test-table [id=9e925792af884fe8bf8e781e8d550d67]) successfully deleted
I20260620 22:02:36.302946 25170 catalog_manager.cc:4841] TS ad6ea6cb53e94c98b87933d07c0b2353 (127.6.241.131:40773): tablet b2fe7e444e0f44858f52cdce36802252 (table test-table [id=9e925792af884fe8bf8e781e8d550d67]) successfully deleted
W20260620 22:02:36.303478 25171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:42743 (error 108)
I20260620 22:02:36.303975 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.303842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42743 (local address 127.0.0.1:38174)
0620 22:02:36.303899 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.303940 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:42743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 22:02:36.304116 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 25380
W20260620 22:02:36.304111 25171 catalog_manager.cc:4584] TS a82029e2aa5643289cde21aed06365ba (127.6.241.129:42743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2fe7e444e0f44858f52cdce36802252: Network error: Client connection negotiation failed: client connection to 127.6.241.129:42743: connect: Connection refused (error 111)
I20260620 22:02:36.309880 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 25514
I20260620 22:02:36.315439 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 25152
2026-06-20T22:02:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2030 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T22:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:36Z Disabled control of system clock
I20260620 22:02:36.341908 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:45757
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:45757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:02:36.420490 25687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:36.420680 25687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:36.420701 25687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:36.421842 25687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:36.421880 25687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:36.421893 25687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:36.421905 25687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:36.421957 25687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:36.423297 25687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:45757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:45757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:36.423527 25687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:36.423796 25687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:36.428439 25693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:36.428488 25695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:36.428525 25692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:36.428646 25687 server_base.cc:1034] running on GCE node
I20260620 22:02:36.428850 25687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:36.429067 25687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:36.430205 25687 hybrid_clock.cc:648] HybridClock initialized: now 1781992956430193 us; error 28 us; skew 500 ppm
I20260620 22:02:36.431457 25687 webserver.cc:460] Webserver started at http://127.6.241.190:40485/ using document root <none> and password file <none>
I20260620 22:02:36.431695 25687 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:36.431742 25687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:36.431905 25687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:36.432893 25687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "1209f1e92133f1dfddc6b00ad0b160b1"
server_key_iv: "3e17712aa902c4a85c824cff48b04ae7"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.433228 25687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "1209f1e92133f1dfddc6b00ad0b160b1"
server_key_iv: "3e17712aa902c4a85c824cff48b04ae7"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.434484 25687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:36.435247 25701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.435436 25687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:36.435518 25687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "1209f1e92133f1dfddc6b00ad0b160b1"
server_key_iv: "3e17712aa902c4a85c824cff48b04ae7"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.435600 25687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:36.461126 25687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:36.461403 25687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:36.461549 25687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:36.465310 25687 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:45757
I20260620 22:02:36.465392 25753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:45757 every 8 connection(s)
I20260620 22:02:36.465689 25687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:36.465926 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 25687
I20260620 22:02:36.466029 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:36.466228 7110 external_mini_cluster.cc:1428] Setting key 3823dbc30b19dbf5f7ec9a20fa9b4a9b
I20260620 22:02:36.466895 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:36.469627 25754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef: Bootstrap starting.
I20260620 22:02:36.470247 25754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:36.470558 25754 log.cc:824] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:36.471387 25754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef: No bootstrap required, opened a new log
I20260620 22:02:36.471583 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.466721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.6.241.190:45757)
0620 22:02:36.466921 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.466928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.466973 (+ 45us) server_negotiation.cc:407] Connection header received
0620 22:02:36.467390 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.467397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.467451 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.467555 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:36.467797 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.468604 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.469716 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.470111 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.470198 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.470904 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:36.470909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:36.470909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:36.471097 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:36.471142 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:36.471152 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:36.471225 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.471316 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.471393 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
I20260620 22:02:36.473232 25754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45757 } }
I20260620 22:02:36.473392 25754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:36.473451 25754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4e6b415a2a4ffeb2ac6fadb6d00fef, State: Initialized, Role: FOLLOWER
I20260620 22:02:36.473562 25754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45757 } }
I20260620 22:02:36.473649 25754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:36.473691 25754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:36.473737 25754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:36.474252 25754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45757 } }
I20260620 22:02:36.474377 25754 leader_election.cc:302] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a4e6b415a2a4ffeb2ac6fadb6d00fef; no voters:
I20260620 22:02:36.474560 25754 leader_election.cc:288] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:36.474647 25759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:36.474803 25759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [term 1 LEADER]: Becoming Leader. State: Replica: 9a4e6b415a2a4ffeb2ac6fadb6d00fef, State: Running, Role: LEADER
I20260620 22:02:36.474831 25754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:36.474889 25759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45757 } }
I20260620 22:02:36.475273 25761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4e6b415a2a4ffeb2ac6fadb6d00fef. Latest consensus state: current_term: 1 leader_uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45757 } } }
I20260620 22:02:36.475262 25760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4e6b415a2a4ffeb2ac6fadb6d00fef" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45757 } } }
I20260620 22:02:36.475363 25761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:36.475371 25760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:36.476112 25768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:36.476553 25768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:36.477985 25768 catalog_manager.cc:1347] Generated new cluster ID: 6bc7a2196d814fa2a3c4545c764b8ee2
I20260620 22:02:36.478040 25768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:36.496047 25768 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:36.496567 25768 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:36.504779 25768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef: Generated new TSK 0
I20260620 22:02:36.504959 25768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:36.510139 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:45757
--builtin_ntp_servers=127.6.241.148:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:02:36.593195 25778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:36.593416 25778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:36.593452 25778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:36.594651 25778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:36.594724 25778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:36.594763 25778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:02:36.596092 25778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:45757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:36.596341 25778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:36.596619 25778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:36.602488 25783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:36.602522 25786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:36.602649 25778 server_base.cc:1034] running on GCE node
W20260620 22:02:36.602509 25784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:36.602902 25778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:36.603174 25778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:36.604331 25778 hybrid_clock.cc:648] HybridClock initialized: now 1781992956604305 us; error 43 us; skew 500 ppm
I20260620 22:02:36.608479 25778 webserver.cc:460] Webserver started at http://127.6.241.129:37975/ using document root <none> and password file <none>
I20260620 22:02:36.608703 25778 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:36.608767 25778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:36.608916 25778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:36.609952 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "6c4d345707ad4499b478cd537b19cf86"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "b88b450a6d9bbebbd9b1a9a9a05e4453"
server_key_iv: "45f4942f7e4f120de1dda390bc8b659f"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.610306 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "6c4d345707ad4499b478cd537b19cf86"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "b88b450a6d9bbebbd9b1a9a9a05e4453"
server_key_iv: "45f4942f7e4f120de1dda390bc8b659f"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.611641 25778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:36.612347 25792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.612501 25778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:36.612599 25778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "6c4d345707ad4499b478cd537b19cf86"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "b88b450a6d9bbebbd9b1a9a9a05e4453"
server_key_iv: "45f4942f7e4f120de1dda390bc8b659f"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.612679 25778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:36.666162 25778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:36.666477 25778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:36.666669 25778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:36.666917 25778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:36.667265 25778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:36.667320 25778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.667361 25778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:36.667389 25778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.672935 25778 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:46193
I20260620 22:02:36.673029 25905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:46193 every 8 connection(s)
I20260620 22:02:36.673329 25778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:02:36.675874 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 25778
I20260620 22:02:36.675980 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:02:36.676167 7110 external_mini_cluster.cc:1428] Setting key 92a16f2047b19491f39b83838a746e79
I20260620 22:02:36.677174 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:45757
--builtin_ntp_servers=127.6.241.148:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:02:36.679224 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.674115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:51109 (local address 127.6.241.190:45757)
0620 22:02:36.674185 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.674187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.674450 (+ 263us) server_negotiation.cc:407] Connection header received
0620 22:02:36.674967 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.674968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.674980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.675064 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:36.675852 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.676575 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.677793 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.677986 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.678063 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.678941 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:36.678942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:36.678943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:36.678948 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:36.678963 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:36.678967 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:36.679036 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.679116 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.679144 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:02:36.679457 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.674121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:45757 (local address 127.6.241.129:51109)
0620 22:02:36.674300 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.674340 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.674836 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.675157 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.675160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.675168 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:36.675743 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.675749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.676704 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.676705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.677699 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.677702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.678069 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.678071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.678218 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.678795 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:02:36.678795 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:36.678824 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:02:36.678845 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:36.679038 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:36.679045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:36.679048 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.679109 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.679208 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260620 22:02:36.679967 25906 heartbeater.cc:347] Connected to a master server at 127.6.241.190:45757
I20260620 22:02:36.680058 25906 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:36.680259 25906 heartbeater.cc:511] Master 127.6.241.190:45757 requested a full tablet report, sending...
I20260620 22:02:36.680825 25718 ts_manager.cc:194] Registered new tserver with Master: 6c4d345707ad4499b478cd537b19cf86 (127.6.241.129:46193)
I20260620 22:02:36.681815 25718 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.129:51109
W20260620 22:02:36.756690 25909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:36.756877 25909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:36.756898 25909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:36.757958 25909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:36.757992 25909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:36.758016 25909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:02:36.759269 25909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:45757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:36.759436 25909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:36.759672 25909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:36.764273 25914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:36.764288 25917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:36.764359 25909 server_base.cc:1034] running on GCE node
W20260620 22:02:36.764288 25915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:36.764727 25909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:36.764968 25909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:36.766124 25909 hybrid_clock.cc:648] HybridClock initialized: now 1781992956766111 us; error 25 us; skew 500 ppm
I20260620 22:02:36.767262 25909 webserver.cc:460] Webserver started at http://127.6.241.130:44895/ using document root <none> and password file <none>
I20260620 22:02:36.767505 25909 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:36.767573 25909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:36.767721 25909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:36.768676 25909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "b0431af124ea42f89515165db2a65c6c"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "c7f29d6cb8d838c745db8c8913af7f93"
server_key_iv: "9807ae18ea56b83f2a5f5f6a6b7b3695"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.769013 25909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "b0431af124ea42f89515165db2a65c6c"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "c7f29d6cb8d838c745db8c8913af7f93"
server_key_iv: "9807ae18ea56b83f2a5f5f6a6b7b3695"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.770246 25909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:36.770977 25923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.771139 25909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.771234 25909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "b0431af124ea42f89515165db2a65c6c"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "c7f29d6cb8d838c745db8c8913af7f93"
server_key_iv: "9807ae18ea56b83f2a5f5f6a6b7b3695"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.771313 25909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:36.783471 25909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:36.783730 25909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:36.783872 25909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:36.784077 25909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:36.784370 25909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:36.784431 25909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.784471 25909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:36.784499 25909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.790122 25909 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:45051
I20260620 22:02:36.790155 26036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:45051 every 8 connection(s)
I20260620 22:02:36.790467 25909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:02:36.792445 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 25909
I20260620 22:02:36.792520 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:02:36.792712 7110 external_mini_cluster.cc:1428] Setting key edd8b74692f212ed6ff1a6a3398555b9
I20260620 22:02:36.793696 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:45757
--builtin_ntp_servers=127.6.241.148:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:02:36.796083 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.791179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:44005 (local address 127.6.241.190:45757)
0620 22:02:36.791224 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.791226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.791418 (+ 192us) server_negotiation.cc:407] Connection header received
0620 22:02:36.791942 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.791944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.791957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.792003 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:36.792715 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.793428 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.794526 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.794869 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.794948 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.795751 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:36.795753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:36.795754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:36.795766 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:36.795784 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:36.795789 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:36.795849 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.795942 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.795969 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:02:36.796185 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.791144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:45757 (local address 127.6.241.130:44005)
0620 22:02:36.791307 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.791337 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.791829 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.792079 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.792085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.792098 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:36.792622 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.792627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.793513 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.793514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.794435 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.794438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.794968 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.794969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.795092 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.795619 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:02:36.795619 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:36.795646 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:02:36.795668 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:36.795855 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:36.795860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:36.795862 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.795922 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.796001 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260620 22:02:36.796648 26037 heartbeater.cc:347] Connected to a master server at 127.6.241.190:45757
I20260620 22:02:36.796751 26037 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:36.796918 26037 heartbeater.cc:511] Master 127.6.241.190:45757 requested a full tablet report, sending...
I20260620 22:02:36.797313 25718 ts_manager.cc:194] Registered new tserver with Master: b0431af124ea42f89515165db2a65c6c (127.6.241.130:45051)
I20260620 22:02:36.797856 25718 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.130:44005
W20260620 22:02:36.874154 26040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:36.874348 26040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:36.874369 26040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:36.875801 26040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:36.875844 26040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:36.875871 26040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:02:36.877274 26040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:45757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:36.877502 26040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:36.877791 26040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:36.882592 26045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:36.882640 26046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:36.882630 26048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:36.882750 26040 server_base.cc:1034] running on GCE node
I20260620 22:02:36.883275 26040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:36.883509 26040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:36.884667 26040 hybrid_clock.cc:648] HybridClock initialized: now 1781992956884656 us; error 24 us; skew 500 ppm
I20260620 22:02:36.885905 26040 webserver.cc:460] Webserver started at http://127.6.241.131:46757/ using document root <none> and password file <none>
I20260620 22:02:36.886134 26040 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:36.886204 26040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:36.886358 26040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:36.887428 26040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "d8fcb4ae21534c3d980927c7e37f3729"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "c629f67f5023ecab12aca6dbd63b1c92"
server_key_iv: "ee2fcbf7c0f1f9bff2b66319d58f2b38"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.887781 26040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "d8fcb4ae21534c3d980927c7e37f3729"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "c629f67f5023ecab12aca6dbd63b1c92"
server_key_iv: "ee2fcbf7c0f1f9bff2b66319d58f2b38"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.889099 26040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:02:36.889757 26054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.889915 26040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:02:36.890009 26040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "d8fcb4ae21534c3d980927c7e37f3729"
format_stamp: "Formatted at 2026-06-20 22:02:36 on dist-test-slave-gl2g"
server_key: "c629f67f5023ecab12aca6dbd63b1c92"
server_key_iv: "ee2fcbf7c0f1f9bff2b66319d58f2b38"
server_key_version: "encryptionkey@0"
I20260620 22:02:36.890089 26040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:36.904055 26040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:36.904335 26040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:36.904479 26040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:36.904681 26040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:36.904984 26040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:36.905037 26040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.905071 26040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:36.905097 26040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:36.910773 26040 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:34785
I20260620 22:02:36.910822 26167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:34785 every 8 connection(s)
I20260620 22:02:36.911109 26040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:02:36.915871 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.911793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:53301 (local address 127.6.241.190:45757)
0620 22:02:36.911858 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.911860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.912056 (+ 196us) server_negotiation.cc:407] Connection header received
0620 22:02:36.912524 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.912525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.912534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.912591 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:36.913181 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.913654 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.914589 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.914792 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.914840 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.915651 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:36.915651 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:36.915652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:36.915655 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:36.915667 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:36.915670 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:36.915710 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.915799 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.915817 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:02:36.916071 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.911769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:45757 (local address 127.6.241.131:53301)
0620 22:02:36.911934 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.911973 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.912443 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.912640 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.912643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.912650 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:36.913118 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.913121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.913728 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.913729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.914510 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.914513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.914857 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.914858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.914962 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.915507 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:02:36.915507 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:36.915532 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:02:36.915552 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:36.915724 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:36.915729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:36.915732 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.915800 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.915872 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260620 22:02:36.916501 26168 heartbeater.cc:347] Connected to a master server at 127.6.241.190:45757
I20260620 22:02:36.916603 26168 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:36.916774 26168 heartbeater.cc:511] Master 127.6.241.190:45757 requested a full tablet report, sending...
I20260620 22:02:36.917141 25718 ts_manager.cc:194] Registered new tserver with Master: d8fcb4ae21534c3d980927c7e37f3729 (127.6.241.131:34785)
I20260620 22:02:36.917515 25718 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.131:53301
I20260620 22:02:36.918824 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 26040
I20260620 22:02:36.918900 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:02:36.919075 7110 external_mini_cluster.cc:1428] Setting key ec03dc557a09c68138868cf1fc1136b8
I20260620 22:02:36.920095 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:02:36.924422 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.921414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.6.241.190:45757)
0620 22:02:36.921487 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.921489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.921597 (+ 108us) server_negotiation.cc:407] Connection header received
0620 22:02:36.921671 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.921672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.921681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.921751 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:36.921907 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.922352 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.923175 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.923363 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.923432 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.924141 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:36.924143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:36.924143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:36.924148 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:36.924160 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:36.924163 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:36.924267 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.924339 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.924348 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:02:36.925846 25718 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:02:36.927459 25718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:02:36.933382 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.929613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:34785 (local address 127.0.0.1:39348)
0620 22:02:36.929956 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.929965 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.930017 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.930185 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.930186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.930187 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.930217 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.930217 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.931176 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.931177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.932383 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.932393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.933107 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.933108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.933181 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.933193 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.933256 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.933308 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260620 22:02:36.933398 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.929792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.6.241.131:34785)
0620 22:02:36.929948 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.929952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.930025 (+ 73us) server_negotiation.cc:407] Connection header received
0620 22:02:36.930082 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.930084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.930098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.930178 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.930272 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.931090 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.932484 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.933019 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.933105 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.933186 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.933282 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.933313 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260620 22:02:36.933539 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.929432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:46193 (local address 127.0.0.1:48588)
0620 22:02:36.929593 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.929628 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.929686 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.930042 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.930043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.930046 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.930112 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.930114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.931507 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.931508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.932316 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.932319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.933303 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.933304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.933363 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.933368 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.933416 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.933460 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":26,"thread_start_us":85,"threads_started":1}
I20260620 22:02:36.933627 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.929432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:45051 (local address 127.0.0.1:40976)
0620 22:02:36.929727 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.929739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.929804 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.930031 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.930034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.930036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.930097 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.930098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.931495 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.931496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.932588 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.932590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.933453 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.933453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.933503 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.933506 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.933534 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.933573 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":35,"thread_start_us":57,"threads_started":1}
I20260620 22:02:36.933928 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.929538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.6.241.130:45051)
0620 22:02:36.929727 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.929732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.929806 (+ 74us) server_negotiation.cc:407] Connection header received
0620 22:02:36.929887 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.929889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.929905 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.930195 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.930268 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.931397 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.932668 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.933360 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.933687 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.933780 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.933806 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.933847 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260620 22:02:36.934386 26102 tablet_service.cc:1467] Processing CreateTablet for tablet 29cb15b2ab1440008efb8c0682444854 (DEFAULT_TABLE table=test-table [id=94527c89b0dd45579733b4d10ac6d19d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:36.934559 25971 tablet_service.cc:1467] Processing CreateTablet for tablet 29cb15b2ab1440008efb8c0682444854 (DEFAULT_TABLE table=test-table [id=94527c89b0dd45579733b4d10ac6d19d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:36.934755 26102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29cb15b2ab1440008efb8c0682444854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:36.934883 25971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29cb15b2ab1440008efb8c0682444854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:36.938062 26183 tablet_bootstrap.cc:492] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: Bootstrap starting.
I20260620 22:02:36.938126 26179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.929538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.6.241.129:46193)
0620 22:02:36.929807 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.929811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.929821 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:02:36.929861 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.929863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.929878 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.929964 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.930309 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.931422 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.932405 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.933118 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.933185 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.933266 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.933603 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.933645 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":85,"threads_started":1}
I20260620 22:02:36.938756 26183 tablet_bootstrap.cc:654] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:36.938766 26182 tablet_bootstrap.cc:492] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: Bootstrap starting.
I20260620 22:02:36.938751 25840 tablet_service.cc:1467] Processing CreateTablet for tablet 29cb15b2ab1440008efb8c0682444854 (DEFAULT_TABLE table=test-table [id=94527c89b0dd45579733b4d10ac6d19d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:36.939069 25840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29cb15b2ab1440008efb8c0682444854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:36.939090 26183 log.cc:824] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:36.939450 26182 tablet_bootstrap.cc:654] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:36.939764 26182 log.cc:824] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:36.939806 26183 tablet_bootstrap.cc:492] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: No bootstrap required, opened a new log
I20260620 22:02:36.939858 26183 ts_tablet_manager.cc:1397] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:02:36.940531 26182 tablet_bootstrap.cc:492] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: No bootstrap required, opened a new log
I20260620 22:02:36.940611 26182 ts_tablet_manager.cc:1397] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:02:36.941916 26186 tablet_bootstrap.cc:492] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86: Bootstrap starting.
I20260620 22:02:36.942468 26186 tablet_bootstrap.cc:654] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:36.942466 26183 raft_consensus.cc:348] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } }
I20260620 22:02:36.942653 26183 raft_consensus.cc:374] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:36.942694 26183 raft_consensus.cc:729] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fcb4ae21534c3d980927c7e37f3729, State: Initialized, Role: FOLLOWER
I20260620 22:02:36.942817 26186 log.cc:824] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:36.942826 26183 consensus_queue.cc:260] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } }
I20260620 22:02:36.943094 26183 ts_tablet_manager.cc:1428] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 22:02:36.943133 26168 heartbeater.cc:503] Master 127.6.241.190:45757 was elected leader, sending a full tablet report...
I20260620 22:02:36.943158 26182 raft_consensus.cc:348] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } }
I20260620 22:02:36.943329 26182 raft_consensus.cc:374] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:36.943372 26182 raft_consensus.cc:729] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0431af124ea42f89515165db2a65c6c, State: Initialized, Role: FOLLOWER
I20260620 22:02:36.943480 26182 consensus_queue.cc:260] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } }
I20260620 22:02:36.943671 26186 tablet_bootstrap.cc:492] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86: No bootstrap required, opened a new log
I20260620 22:02:36.943735 26182 ts_tablet_manager.cc:1428] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:02:36.943744 26186 ts_tablet_manager.cc:1397] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:02:36.943792 26037 heartbeater.cc:503] Master 127.6.241.190:45757 was elected leader, sending a full tablet report...
I20260620 22:02:36.945899 26186 raft_consensus.cc:348] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } }
I20260620 22:02:36.946014 26186 raft_consensus.cc:374] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:36.946034 26186 raft_consensus.cc:729] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4d345707ad4499b478cd537b19cf86, State: Initialized, Role: FOLLOWER
I20260620 22:02:36.946101 26186 consensus_queue.cc:260] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } }
I20260620 22:02:36.946278 26186 ts_tablet_manager.cc:1428] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 22:02:36.946362 25906 heartbeater.cc:503] Master 127.6.241.190:45757 was elected leader, sending a full tablet report...
I20260620 22:02:36.963371 26190 raft_consensus.cc:482] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:36.963443 26190 raft_consensus.cc:504] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } }
I20260620 22:02:36.963709 26190 leader_election.cc:288] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0431af124ea42f89515165db2a65c6c (127.6.241.130:45051), d8fcb4ae21534c3d980927c7e37f3729 (127.6.241.131:34785)
I20260620 22:02:36.967259 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.963949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:49183 (local address 127.6.241.131:34785)
0620 22:02:36.964004 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.964006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.964137 (+ 131us) server_negotiation.cc:407] Connection header received
0620 22:02:36.964166 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.964167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.964173 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.964247 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.964360 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.965027 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.966363 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.966919 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.966994 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.967001 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.967161 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.967196 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:02:36.967259 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.963924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:33797 (local address 127.6.241.130:45051)
0620 22:02:36.963966 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.963968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.963976 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:02:36.963998 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.963999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.964009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.964070 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:36.964241 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.964976 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.966365 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.966926 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.966994 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.967001 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.967162 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.967200 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:02:36.967275 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.963845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:45051 (local address 127.6.241.129:33797)
0620 22:02:36.963883 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.963895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.963937 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.964098 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.964099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.964100 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.964167 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.964168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.965054 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.965055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.966276 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.966278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.967004 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.967005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.967079 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.967084 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.967143 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.967194 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":26}
I20260620 22:02:36.967291 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.963886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:34785 (local address 127.6.241.129:49183)
0620 22:02:36.964034 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.964048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:36.964113 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:36.964250 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:36.964251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:36.964252 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:36.964303 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.964303 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.965093 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.965093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.966286 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:36.966288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.967005 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.967006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:36.967089 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.967092 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:36.967143 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:02:36.967194 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":9,"thread_start_us":83,"threads_started":1}
I20260620 22:02:36.967585 25991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29cb15b2ab1440008efb8c0682444854" candidate_uuid: "6c4d345707ad4499b478cd537b19cf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0431af124ea42f89515165db2a65c6c" is_pre_election: true
I20260620 22:02:36.967657 26122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29cb15b2ab1440008efb8c0682444854" candidate_uuid: "6c4d345707ad4499b478cd537b19cf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8fcb4ae21534c3d980927c7e37f3729" is_pre_election: true
I20260620 22:02:36.967682 25991 raft_consensus.cc:2454] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c4d345707ad4499b478cd537b19cf86 in term 0.
I20260620 22:02:36.967780 26122 raft_consensus.cc:2454] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c4d345707ad4499b478cd537b19cf86 in term 0.
I20260620 22:02:36.967936 25793 leader_election.cc:302] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4d345707ad4499b478cd537b19cf86, b0431af124ea42f89515165db2a65c6c; no voters:
I20260620 22:02:36.968068 26190 raft_consensus.cc:2781] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:02:36.968122 26190 raft_consensus.cc:482] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:02:36.968138 26190 raft_consensus.cc:3037] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:36.968701 26190 raft_consensus.cc:504] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } }
I20260620 22:02:36.968827 26190 leader_election.cc:288] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [CANDIDATE]: Term 1 election: Requested vote from peers b0431af124ea42f89515165db2a65c6c (127.6.241.130:45051), d8fcb4ae21534c3d980927c7e37f3729 (127.6.241.131:34785)
I20260620 22:02:36.968945 25991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29cb15b2ab1440008efb8c0682444854" candidate_uuid: "6c4d345707ad4499b478cd537b19cf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0431af124ea42f89515165db2a65c6c"
I20260620 22:02:36.969012 26122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29cb15b2ab1440008efb8c0682444854" candidate_uuid: "6c4d345707ad4499b478cd537b19cf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8fcb4ae21534c3d980927c7e37f3729"
I20260620 22:02:36.969010 25991 raft_consensus.cc:3037] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:36.969094 26122 raft_consensus.cc:3037] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:36.969664 25991 raft_consensus.cc:2454] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c4d345707ad4499b478cd537b19cf86 in term 1.
I20260620 22:02:36.969780 26122 raft_consensus.cc:2454] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c4d345707ad4499b478cd537b19cf86 in term 1.
I20260620 22:02:36.969826 25793 leader_election.cc:302] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4d345707ad4499b478cd537b19cf86, b0431af124ea42f89515165db2a65c6c; no voters:
I20260620 22:02:36.969949 26190 raft_consensus.cc:2781] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:36.970116 26190 raft_consensus.cc:686] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [term 1 LEADER]: Becoming Leader. State: Replica: 6c4d345707ad4499b478cd537b19cf86, State: Running, Role: LEADER
I20260620 22:02:36.970211 26190 consensus_queue.cc:237] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } }
I20260620 22:02:36.970932 25718 catalog_manager.cc:5495] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c4d345707ad4499b478cd537b19cf86 (127.6.241.129). New cstate: current_term: 1 leader_uuid: "6c4d345707ad4499b478cd537b19cf86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c4d345707ad4499b478cd537b19cf86" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260620 22:02:36.978101 26179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.975989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.6.241.129:46193)
0620 22:02:36.976038 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:36.976040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:36.976048 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:02:36.976116 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:36.976117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:36.976130 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:36.976186 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:02:36.976302 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.976727 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.977461 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:36.977631 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:36.977683 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:36.977803 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:02:36.977873 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:02:36.977937 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:36.977982 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 22:02:36.977990 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:02:36.979660 26122 raft_consensus.cc:1261] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [term 1 FOLLOWER]: Refusing update from remote peer 6c4d345707ad4499b478cd537b19cf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:36.979660 25991 raft_consensus.cc:1261] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [term 1 FOLLOWER]: Refusing update from remote peer 6c4d345707ad4499b478cd537b19cf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:36.980027 26192 consensus_queue.cc:1035] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8fcb4ae21534c3d980927c7e37f3729" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:36.980132 26190 consensus_queue.cc:1035] T 29cb15b2ab1440008efb8c0682444854 P 6c4d345707ad4499b478cd537b19cf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0431af124ea42f89515165db2a65c6c" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:36.982105 26195 mvcc.cc:205] Tried to move back new op lower bound from 7299043151786299392 to 7299043151750414336. Current Snapshot: MvccSnapshot[applied={T|T < 7299043151786299392}]
I20260620 22:02:36.982893 26197 mvcc.cc:205] Tried to move back new op lower bound from 7299043151786299392 to 7299043151750414336. Current Snapshot: MvccSnapshot[applied={T|T < 7299043151786299392}]
I20260620 22:02:36.983163 26196 mvcc.cc:205] Tried to move back new op lower bound from 7299043151786299392 to 7299043151750414336. Current Snapshot: MvccSnapshot[applied={T|T < 7299043151786299392}]
I20260620 22:02:36.984279 25716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:36.984359 25716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:36.985045 25716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a4e6b415a2a4ffeb2ac6fadb6d00fef: Sending DeleteTablet for 3 replicas of tablet 29cb15b2ab1440008efb8c0682444854
I20260620 22:02:36.985364 26102 tablet_service.cc:1514] Processing DeleteTablet for tablet 29cb15b2ab1440008efb8c0682444854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:36 UTC) from {username='slave'} at 127.0.0.1:39348
I20260620 22:02:36.985396 25971 tablet_service.cc:1514] Processing DeleteTablet for tablet 29cb15b2ab1440008efb8c0682444854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:36 UTC) from {username='slave'} at 127.0.0.1:40976
I20260620 22:02:36.985379 25840 tablet_service.cc:1514] Processing DeleteTablet for tablet 29cb15b2ab1440008efb8c0682444854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:36 UTC) from {username='slave'} at 127.0.0.1:48588
I20260620 22:02:36.985759 26206 tablet_replica.cc:330] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: stopping tablet replica
I20260620 22:02:36.985846 26206 raft_consensus.cc:2229] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:02:36.985918 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 25778
I20260620 22:02:36.985945 26206 raft_consensus.cc:2258] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:36.987445 26205 tablet_replica.cc:330] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: stopping tablet replica
I20260620 22:02:36.987543 26205 raft_consensus.cc:2229] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:02:36.987641 26205 raft_consensus.cc:2258] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:36.988286 26205 ts_tablet_manager.cc:1905] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:36.988467 26206 ts_tablet_manager.cc:1905] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:36.989758 26205 ts_tablet_manager.cc:1918] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:36.989840 26205 log.cc:1196] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/29cb15b2ab1440008efb8c0682444854
I20260620 22:02:36.990083 26205 ts_tablet_manager.cc:1939] T 29cb15b2ab1440008efb8c0682444854 P d8fcb4ae21534c3d980927c7e37f3729: Deleting consensus metadata
I20260620 22:02:36.990273 26206 ts_tablet_manager.cc:1918] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:36.990347 26206 log.cc:1196] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/29cb15b2ab1440008efb8c0682444854
I20260620 22:02:36.990485 25704 catalog_manager.cc:4841] TS d8fcb4ae21534c3d980927c7e37f3729 (127.6.241.131:34785): tablet 29cb15b2ab1440008efb8c0682444854 (table test-table [id=94527c89b0dd45579733b4d10ac6d19d]) successfully deleted
I20260620 22:02:36.990581 26206 ts_tablet_manager.cc:1939] T 29cb15b2ab1440008efb8c0682444854 P b0431af124ea42f89515165db2a65c6c: Deleting consensus metadata
I20260620 22:02:36.990983 25702 catalog_manager.cc:4841] TS b0431af124ea42f89515165db2a65c6c (127.6.241.130:45051): tablet 29cb15b2ab1440008efb8c0682444854 (table test-table [id=94527c89b0dd45579733b4d10ac6d19d]) successfully deleted
W20260620 22:02:36.992269 25704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:46193 (error 108)
I20260620 22:02:36.992453 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 25909
I20260620 22:02:36.993027 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:36.992604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:46193 (local address 127.0.0.1:48604)
0620 22:02:36.992947 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:36.992990 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322}
W20260620 22:02:36.993134 25704 catalog_manager.cc:4584] TS 6c4d345707ad4499b478cd537b19cf86 (127.6.241.129:46193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29cb15b2ab1440008efb8c0682444854: Network error: Client connection negotiation failed: client connection to 127.6.241.129:46193: connect: Connection refused (error 111)
I20260620 22:02:36.997658 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 26040
I20260620 22:02:37.001794 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 25687
2026-06-20T22:02:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (685 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:37 dist-test-slave-gl2g krb5kdc[26211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:37 dist-test-slave-gl2g krb5kdc[26211](info): set up 2 sockets
Jun 20 22:02:37 dist-test-slave-gl2g krb5kdc[26211](info): commencing operation
krb5kdc: starting...
W20260620 22:02:38.026233 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:38 dist-test-slave-gl2g krb5kdc[26211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:02:38.136441 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40467
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:40467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:38.213642 26224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:38.213826 26224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:38.213876 26224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:38.215003 26224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:38.215044 26224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:38.215057 26224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:38.215070 26224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:38.215081 26224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:38.216442 26224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:40467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:40467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:38.216615 26224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:38.216764 26224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:02:38.221402 26224 server_base.cc:1034] running on GCE node
W20260620 22:02:38.221375 26230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:38.221375 26232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:38.221362 26229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:38.221781 26224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:38.222108 26224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:38.223282 26224 hybrid_clock.cc:648] HybridClock initialized: now 1781992958223257 us; error 39 us; skew 500 ppm
Jun 20 22:02:38 dist-test-slave-gl2g krb5kdc[26211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:38.225623 26224 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:02:38.226068 26224 webserver.cc:460] Webserver started at http://127.6.241.190:45039/ using document root <none> and password file <none>
I20260620 22:02:38.226292 26224 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:38.226377 26224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:38.226507 26224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:38.227520 26224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "8e9a5f00b7f146a3958039071ab7459b"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "f65c70c9aa0ff059315b17d72d3117e9"
server_key_iv: "6a26a8785570a5b8d119906639aaf306"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.227860 26224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "8e9a5f00b7f146a3958039071ab7459b"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "f65c70c9aa0ff059315b17d72d3117e9"
server_key_iv: "6a26a8785570a5b8d119906639aaf306"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.229110 26224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:02:38.229780 26239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.229967 26224 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:38.230082 26224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "8e9a5f00b7f146a3958039071ab7459b"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "f65c70c9aa0ff059315b17d72d3117e9"
server_key_iv: "6a26a8785570a5b8d119906639aaf306"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.230160 26224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:38.258167 26224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:38.259980 26224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:38.260146 26224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:38.264145 26224 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:40467
I20260620 22:02:38.264179 26291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:40467 every 8 connection(s)
I20260620 22:02:38.264540 26224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:38.265362 26292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:38.267653 26292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b: Bootstrap starting.
I20260620 22:02:38.268232 26292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:38.268524 26292 log.cc:824] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:38.269191 26292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b: No bootstrap required, opened a new log
I20260620 22:02:38.270767 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 26224
I20260620 22:02:38.270879 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:38.270788 26292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9a5f00b7f146a3958039071ab7459b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40467 } }
I20260620 22:02:38.270999 26292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:38.271067 26292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9a5f00b7f146a3958039071ab7459b, State: Initialized, Role: FOLLOWER
I20260620 22:02:38.271107 7110 external_mini_cluster.cc:1428] Setting key dc765ae38025da731b713dfd071b3dc3
I20260620 22:02:38.271210 26292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9a5f00b7f146a3958039071ab7459b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40467 } }
I20260620 22:02:38.271288 26292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:38.271334 26292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:38.271380 26292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:38.272230 26292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9a5f00b7f146a3958039071ab7459b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40467 } }
I20260620 22:02:38.272315 26292 leader_election.cc:302] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e9a5f00b7f146a3958039071ab7459b; no voters:
I20260620 22:02:38.272485 26292 leader_election.cc:288] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:38.272531 26297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:38.272667 26297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [term 1 LEADER]: Becoming Leader. State: Replica: 8e9a5f00b7f146a3958039071ab7459b, State: Running, Role: LEADER
I20260620 22:02:38.272733 26292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:38.272753 26297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9a5f00b7f146a3958039071ab7459b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40467 } }
I20260620 22:02:38.273054 26297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e9a5f00b7f146a3958039071ab7459b. Latest consensus state: current_term: 1 leader_uuid: "8e9a5f00b7f146a3958039071ab7459b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9a5f00b7f146a3958039071ab7459b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40467 } } }
I20260620 22:02:38.273132 26297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:38.273065 26299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e9a5f00b7f146a3958039071ab7459b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9a5f00b7f146a3958039071ab7459b" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 40467 } } }
I20260620 22:02:38.273317 26302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:38.273320 26299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:38.273797 26302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:38.275604 26302 catalog_manager.cc:1347] Generated new cluster ID: 67584ed41c7d4fd6bbe4e0c3c539ea96
I20260620 22:02:38.275656 26302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:02:38 dist-test-slave-gl2g krb5kdc[26211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:38.282675 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.271523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.6.241.190:40467)
0620 22:02:38.271695 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.271705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.271738 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:02:38.271850 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.271856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.271893 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.272022 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:38.272804 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.274193 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.275298 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.275874 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.276012 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.278581 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:38.278585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:38.278586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:38.278622 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:38.281094 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.281463 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.281466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.281466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.281581 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.281779 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.281780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.281780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.282078 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:38.282236 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.282305 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.282433 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260620 22:02:38.284334 26302 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:38.284816 26302 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:38.288250 26302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b: Generated new TSK 0
I20260620 22:02:38.288439 26302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.241.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
Entry for principal kudu/127.6.241.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab.
I20260620 22:02:38.311455 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40467
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:38.389060 26318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:38.389288 26318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:38.389334 26318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:38.390468 26318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:38.390537 26318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:38.390581 26318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:02:38.392026 26318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.129.keytab
--principal=kudu/127.6.241.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:38.392287 26318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:38.392506 26318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:38.398334 26326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:38.398458 26318 server_base.cc:1034] running on GCE node
W20260620 22:02:38.398345 26323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:38.398345 26324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:38.398737 26318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:38.398996 26318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:38.400149 26318 hybrid_clock.cc:648] HybridClock initialized: now 1781992958400130 us; error 35 us; skew 500 ppm
Jun 20 22:02:38 dist-test-slave-gl2g krb5kdc[26211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:38.402540 26318 init.cc:377] Logged in from keytab as kudu/127.6.241.129@KRBTEST.COM (short username kudu)
I20260620 22:02:38.403028 26318 webserver.cc:460] Webserver started at http://127.6.241.129:43751/ using document root <none> and password file <none>
I20260620 22:02:38.403196 26318 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:38.403235 26318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:38.403328 26318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:38.404244 26318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "9e48c35195db41a7af9a551d4da954dd"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "cdc827790af35cb3869072613019dfc9"
server_key_iv: "82aee32b2ee5918d246fed1e92ee8aa8"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.404543 26318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "9e48c35195db41a7af9a551d4da954dd"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "cdc827790af35cb3869072613019dfc9"
server_key_iv: "82aee32b2ee5918d246fed1e92ee8aa8"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.405735 26318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:02:38.406374 26333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.406562 26318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:38.406685 26318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "9e48c35195db41a7af9a551d4da954dd"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "cdc827790af35cb3869072613019dfc9"
server_key_iv: "82aee32b2ee5918d246fed1e92ee8aa8"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.406764 26318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:38.417110 26318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:38.419440 26318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:38.419598 26318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:38.419812 26318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:38.420110 26318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:38.420163 26318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.420204 26318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:38.420231 26318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.426096 26318 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:44677
I20260620 22:02:38.426144 26446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:44677 every 8 connection(s)
I20260620 22:02:38.426451 26318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:02:38 dist-test-slave-gl2g krb5kdc[26211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.129@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:38.434855 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.427128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:43357 (local address 127.6.241.190:40467)
0620 22:02:38.427195 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.427197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.427402 (+ 205us) server_negotiation.cc:407] Connection header received
0620 22:02:38.427844 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.427845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.427855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.427924 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:38.428754 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.429245 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.430167 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.430352 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.430402 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.432490 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:38.432492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:38.432492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:38.432498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:38.433657 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.434028 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.434029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.434029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.434077 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.434285 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.434286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.434286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.434378 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:38.434424 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.434732 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.434794 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:02:38.435021 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.427122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40467 (local address 127.6.241.129:43357)
0620 22:02:38.427266 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:38.427312 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:38.427765 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:38.427975 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:38.427978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:38.428504 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:38.428682 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.428686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.429332 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.429333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.430092 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.430094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.430424 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.430425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.430526 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.431069 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:38.431077 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:38.432406 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:38.433738 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:38.433744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:38.433748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:38.433961 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:38.434135 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:38.434135 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:38.434136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:38.434233 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:38.434431 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:38.434435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:38.434538 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 22:02:38.434715 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 22:02:38.434834 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260620 22:02:38.435518 26447 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40467
I20260620 22:02:38.435609 26447 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:38.435788 26447 heartbeater.cc:511] Master 127.6.241.190:40467 requested a full tablet report, sending...
I20260620 22:02:38.435835 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 26318
I20260620 22:02:38.435956 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:02:38.436177 7110 external_mini_cluster.cc:1428] Setting key e7e20d5320d97699acba584b1a33f5e3
I20260620 22:02:38.436385 26256 ts_manager.cc:194] Registered new tserver with Master: 9e48c35195db41a7af9a551d4da954dd (127.6.241.129:44677)
I20260620 22:02:38.437317 26256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.129@KRBTEST.COM'} at 127.6.241.129:43357
WARNING: no policy specified for kudu/127.6.241.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
Entry for principal kudu/127.6.241.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab.
I20260620 22:02:38.459321 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40467
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:38.538477 26452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:38.538754 26452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:38.538792 26452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:38.539909 26452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:38.539978 26452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:38.540022 26452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:02:38.541524 26452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.130.keytab
--principal=kudu/127.6.241.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:38.541813 26452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:38.542037 26452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:38.546833 26458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:38.546970 26452 server_base.cc:1034] running on GCE node
W20260620 22:02:38.546895 26457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:38.546833 26460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:38.547267 26452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:38.547515 26452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:38.548676 26452 hybrid_clock.cc:648] HybridClock initialized: now 1781992958548663 us; error 24 us; skew 500 ppm
Jun 20 22:02:38 dist-test-slave-gl2g krb5kdc[26211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:38.551152 26452 init.cc:377] Logged in from keytab as kudu/127.6.241.130@KRBTEST.COM (short username kudu)
I20260620 22:02:38.551568 26452 webserver.cc:460] Webserver started at http://127.6.241.130:45337/ using document root <none> and password file <none>
I20260620 22:02:38.551805 26452 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:38.551872 26452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:38.552027 26452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:38.553071 26452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "7fb566df7d8b466c957033cd12cd832e"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "6b961f38ae9b4f66765ffcd171033b2d"
server_key_iv: "3151369e73d80137511ff766b2570f60"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.553429 26452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "7fb566df7d8b466c957033cd12cd832e"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "6b961f38ae9b4f66765ffcd171033b2d"
server_key_iv: "3151369e73d80137511ff766b2570f60"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.554831 26452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:02:38.555549 26467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.555713 26452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:38.555810 26452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "7fb566df7d8b466c957033cd12cd832e"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "6b961f38ae9b4f66765ffcd171033b2d"
server_key_iv: "3151369e73d80137511ff766b2570f60"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.555889 26452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:38.579303 26452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:38.581132 26452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:38.581300 26452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:38.581532 26452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:38.581844 26452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:38.581899 26452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.581950 26452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:38.581979 26452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.587687 26452 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:46085
I20260620 22:02:38.587738 26580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:46085 every 8 connection(s)
I20260620 22:02:38.588047 26452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:02:38.593276 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 26452
I20260620 22:02:38.593413 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:02:38.593602 7110 external_mini_cluster.cc:1428] Setting key 41bc351284b1654c5c75d6fb5b291107
Jun 20 22:02:38 dist-test-slave-gl2g krb5kdc[26211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.130@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:38.597577 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.588792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:53879 (local address 127.6.241.190:40467)
0620 22:02:38.588860 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.588862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.589044 (+ 182us) server_negotiation.cc:407] Connection header received
0620 22:02:38.589511 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.589512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.589521 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.589578 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:38.590507 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.591046 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.591997 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.592180 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.592231 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.594486 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:38.594488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:38.594488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:38.594496 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:38.596089 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.596630 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.596631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.596632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.596683 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.596974 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.596975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.596975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.597072 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:38.597122 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.597449 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.597513 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:02:38.597885 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.588760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40467 (local address 127.6.241.130:53879)
0620 22:02:38.588914 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:38.588954 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:38.589428 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:38.589630 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:38.589633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:38.590195 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:38.590385 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.590390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.591125 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.591126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.591921 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.591923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.592253 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.592253 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.592353 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.592769 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:38.592777 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:38.594378 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:38.596176 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:38.596184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:38.596187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:38.596549 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:38.596752 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:38.596753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:38.596754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:38.596907 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:38.597120 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:38.597126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:38.597305 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 22:02:38.597435 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 22:02:38.597611 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260620 22:02:38.598456 26581 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40467
I20260620 22:02:38.598564 26581 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:38.598794 26581 heartbeater.cc:511] Master 127.6.241.190:40467 requested a full tablet report, sending...
I20260620 22:02:38.599261 26256 ts_manager.cc:194] Registered new tserver with Master: 7fb566df7d8b466c957033cd12cd832e (127.6.241.130:46085)
I20260620 22:02:38.599646 26256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.130@KRBTEST.COM'} at 127.6.241.130:53879
WARNING: no policy specified for kudu/127.6.241.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
Entry for principal kudu/127.6.241.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab.
I20260620 22:02:38.617358 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40467
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:38.695271 26586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:38.695472 26586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:38.695495 26586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:38.696660 26586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:38.696702 26586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:38.696758 26586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:02:38.698182 26586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.131.keytab
--principal=kudu/127.6.241.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:38.698426 26586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:38.698669 26586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:38.703359 26591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:38.703346 26594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:38.703378 26592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:38.703729 26586 server_base.cc:1034] running on GCE node
I20260620 22:02:38.703907 26586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:38.704092 26586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:38.705226 26586 hybrid_clock.cc:648] HybridClock initialized: now 1781992958705209 us; error 31 us; skew 500 ppm
Jun 20 22:02:38 dist-test-slave-gl2g krb5kdc[26211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:38.707741 26586 init.cc:377] Logged in from keytab as kudu/127.6.241.131@KRBTEST.COM (short username kudu)
I20260620 22:02:38.708177 26586 webserver.cc:460] Webserver started at http://127.6.241.131:43335/ using document root <none> and password file <none>
I20260620 22:02:38.708389 26586 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:38.708434 26586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:38.708580 26586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:38.709661 26586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "d0e0c56fd9f34ee5941a153a13936d53"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "a38892eed6f0c81945a2a5e7433a8247"
server_key_iv: "44ac0f3a98eea63ba62e64e74a119644"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.710011 26586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "d0e0c56fd9f34ee5941a153a13936d53"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "a38892eed6f0c81945a2a5e7433a8247"
server_key_iv: "44ac0f3a98eea63ba62e64e74a119644"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.711444 26586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:38.712139 26601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.712320 26586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:38.712417 26586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "d0e0c56fd9f34ee5941a153a13936d53"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "a38892eed6f0c81945a2a5e7433a8247"
server_key_iv: "44ac0f3a98eea63ba62e64e74a119644"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.712498 26586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:38.741832 26586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:38.743800 26586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:38.743956 26586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:38.744180 26586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:38.744485 26586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:38.744541 26586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.744582 26586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:38.744611 26586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:38.750311 26586 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:35851
I20260620 22:02:38.750356 26714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:35851 every 8 connection(s)
I20260620 22:02:38.750708 26586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:02:38.751878 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 26586
I20260620 22:02:38.751997 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:02:38.752169 7110 external_mini_cluster.cc:1428] Setting key 89a2b8c4fcdae2336f888fcd6910a86d
Jun 20 22:02:38 dist-test-slave-gl2g krb5kdc[26211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.131@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:38.759740 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.751390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:50527 (local address 127.6.241.190:40467)
0620 22:02:38.751430 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.751432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.751615 (+ 183us) server_negotiation.cc:407] Connection header received
0620 22:02:38.752239 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.752240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.752254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.752330 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:38.753444 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.753923 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.754938 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.755115 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.755168 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.757061 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:38.757062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:38.757062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:38.757069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:38.758534 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.758934 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.758934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.758935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.758985 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.759160 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.759160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.759161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.759253 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:38.759310 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.759609 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.759679 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:02:38.759933 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.751343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:40467 (local address 127.6.241.131:50527)
0620 22:02:38.751497 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:38.751537 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:38.752138 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:38.752434 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:38.752438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:38.753170 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:38.753356 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.753362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.753996 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.753996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.754866 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.754868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.755184 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.755185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.755277 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.755687 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:02:38.755694 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:38.756979 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:38.758636 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:38.758641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:38.758643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:38.758863 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:38.759028 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:02:38.759029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:02:38.759029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:02:38.759112 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:02:38.759310 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:38.759314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:38.759437 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 22:02:38.759570 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 22:02:38.759702 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260620 22:02:38.760417 26715 heartbeater.cc:347] Connected to a master server at 127.6.241.190:40467
I20260620 22:02:38.760507 26715 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:38.760636 26715 heartbeater.cc:511] Master 127.6.241.190:40467 requested a full tablet report, sending...
I20260620 22:02:38.761013 26256 ts_manager.cc:194] Registered new tserver with Master: d0e0c56fd9f34ee5941a153a13936d53 (127.6.241.131:35851)
I20260620 22:02:38.761370 26256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.241.131@KRBTEST.COM'} at 127.6.241.131:50527
I20260620 22:02:38.763761 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:02:38.771144 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.765440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.6.241.190:40467)
0620 22:02:38.765515 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.765517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.765586 (+ 69us) server_negotiation.cc:407] Connection header received
0620 22:02:38.765657 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.765658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.765667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.765720 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:38.766253 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.766794 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.767538 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.767738 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.767820 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.768857 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:38.768858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:38.768859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:38.768865 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:38.769932 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.770266 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.770267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.770267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.770319 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:38.770533 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:38.770534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:38.770534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:38.770733 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:38.770851 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.770977 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.771049 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 22:02:38.772513 26256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:02:38.774255 26256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:02:38.780319 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.776434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:46085 (local address 127.0.0.1:37956)
0620 22:02:38.776525 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:38.776534 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:38.776572 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:38.777019 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:38.777019 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:38.777022 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:38.777076 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.777078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.778018 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.778019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.779198 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.779200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.780008 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.780009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.780085 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.780105 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 22:02:38.780193 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:02:38.780243 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":39,"threads_started":1}
I20260620 22:02:38.780330 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.776705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.6.241.130:46085)
0620 22:02:38.776862 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.776867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.776876 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:02:38.776918 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.776919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.776931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.777019 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:38.777157 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.777917 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.779295 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.779907 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.780004 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.780061 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.780212 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.780250 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260620 22:02:38.780483 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.776457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.6.241.129:44677)
0620 22:02:38.776734 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.776737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.776745 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:02:38.776777 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.776778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.776790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.776878 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:38.777064 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.778203 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.779361 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.779994 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.780064 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.780112 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.780320 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.780369 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":79,"threads_started":1}
I20260620 22:02:38.780730 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.776258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:44677 (local address 127.0.0.1:50104)
0620 22:02:38.776516 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:38.776526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:38.776581 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:38.776892 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:38.776895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:38.776898 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:38.776969 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.776971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.778318 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.778318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.779238 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.779241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.780148 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.780150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.780224 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.780228 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:02:38.780611 (+ 383us) client_negotiation.cc:240] Negotiation successful
0620 22:02:38.780657 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":2,"thread_start_us":45,"threads_started":1}
I20260620 22:02:38.780972 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.776331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.6.241.131:35851)
0620 22:02:38.777003 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.777007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.777016 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:02:38.777049 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.777051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.777065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.777306 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:38.777362 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.778347 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.779621 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.780313 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.780706 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.780767 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.780879 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.780917 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":77,"threads_started":1}
I20260620 22:02:38.780968 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.776258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:35851 (local address 127.0.0.1:40706)
0620 22:02:38.776403 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:38.776440 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:38.776498 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:38.777197 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:38.777198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:38.777200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:38.777235 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.777236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.778448 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.778449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.779542 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.779544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.780724 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.780725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.780795 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.780799 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:02:38.780859 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:02:38.780895 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":53,"threads_started":1}
I20260620 22:02:38.781026 26515 tablet_service.cc:1467] Processing CreateTablet for tablet 3391ddf4c36c42a4a92b728d64908552 (DEFAULT_TABLE table=test-table [id=7a364aff3df24647a7d0e13303e22446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:38.781387 26515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3391ddf4c36c42a4a92b728d64908552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:38.784457 26729 tablet_bootstrap.cc:492] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: Bootstrap starting.
I20260620 22:02:38.784904 26649 tablet_service.cc:1467] Processing CreateTablet for tablet 3391ddf4c36c42a4a92b728d64908552 (DEFAULT_TABLE table=test-table [id=7a364aff3df24647a7d0e13303e22446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:38.784977 26381 tablet_service.cc:1467] Processing CreateTablet for tablet 3391ddf4c36c42a4a92b728d64908552 (DEFAULT_TABLE table=test-table [id=7a364aff3df24647a7d0e13303e22446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:38.785113 26729 tablet_bootstrap.cc:654] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:38.785213 26649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3391ddf4c36c42a4a92b728d64908552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:38.785320 26381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3391ddf4c36c42a4a92b728d64908552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:38.785681 26729 log.cc:824] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:38.786492 26729 tablet_bootstrap.cc:492] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: No bootstrap required, opened a new log
I20260620 22:02:38.786568 26729 ts_tablet_manager.cc:1397] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:02:38.788002 26731 tablet_bootstrap.cc:492] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd: Bootstrap starting.
I20260620 22:02:38.788682 26731 tablet_bootstrap.cc:654] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:38.788700 26732 tablet_bootstrap.cc:492] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: Bootstrap starting.
I20260620 22:02:38.788758 26729 raft_consensus.cc:348] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } }
I20260620 22:02:38.788939 26729 raft_consensus.cc:374] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:38.789000 26729 raft_consensus.cc:729] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb566df7d8b466c957033cd12cd832e, State: Initialized, Role: FOLLOWER
I20260620 22:02:38.789119 26729 consensus_queue.cc:260] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } }
I20260620 22:02:38.789383 26729 ts_tablet_manager.cc:1428] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:02:38.789386 26732 tablet_bootstrap.cc:654] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:38.789501 26581 heartbeater.cc:503] Master 127.6.241.190:40467 was elected leader, sending a full tablet report...
I20260620 22:02:38.789474 26731 log.cc:824] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:38.789963 26732 log.cc:824] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:38.790349 26731 tablet_bootstrap.cc:492] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd: No bootstrap required, opened a new log
I20260620 22:02:38.790441 26731 ts_tablet_manager.cc:1397] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:02:38.791183 26732 tablet_bootstrap.cc:492] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: No bootstrap required, opened a new log
I20260620 22:02:38.791239 26732 ts_tablet_manager.cc:1397] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:02:38.792562 26731 raft_consensus.cc:348] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } }
I20260620 22:02:38.792656 26732 raft_consensus.cc:348] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } }
I20260620 22:02:38.792730 26731 raft_consensus.cc:374] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:38.792778 26732 raft_consensus.cc:374] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:38.792778 26731 raft_consensus.cc:729] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e48c35195db41a7af9a551d4da954dd, State: Initialized, Role: FOLLOWER
I20260620 22:02:38.792817 26732 raft_consensus.cc:729] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e0c56fd9f34ee5941a153a13936d53, State: Initialized, Role: FOLLOWER
I20260620 22:02:38.792896 26731 consensus_queue.cc:260] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } }
I20260620 22:02:38.792922 26732 consensus_queue.cc:260] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } }
I20260620 22:02:38.793181 26447 heartbeater.cc:503] Master 127.6.241.190:40467 was elected leader, sending a full tablet report...
I20260620 22:02:38.793192 26732 ts_tablet_manager.cc:1428] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:02:38.793217 26715 heartbeater.cc:503] Master 127.6.241.190:40467 was elected leader, sending a full tablet report...
I20260620 22:02:38.793432 26731 ts_tablet_manager.cc:1428] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:02:38.834527 26733 raft_consensus.cc:482] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:38.834699 26733 raft_consensus.cc:504] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } }
I20260620 22:02:38.835039 26733 leader_election.cc:288] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e48c35195db41a7af9a551d4da954dd (127.6.241.129:44677), d0e0c56fd9f34ee5941a153a13936d53 (127.6.241.131:35851)
W20260620 22:02:38.838523 26582 tablet.cc:2369] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:02:38.839064 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.835219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:44677 (local address 127.6.241.130:45387)
0620 22:02:38.835308 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:38.835321 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:38.835384 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:38.835583 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:38.835584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:38.835586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:38.835682 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.835684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.836586 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.836586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.837985 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.837988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.838769 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.838770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.838851 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.838855 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:02:38.838938 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:02:38.838993 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":88}
I20260620 22:02:38.839076 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.835275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:47131 (local address 127.6.241.131:35851)
0620 22:02:38.835335 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.835337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.835477 (+ 140us) server_negotiation.cc:407] Connection header received
0620 22:02:38.835510 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.835511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.835520 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.835573 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:38.835758 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.836510 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.838082 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.838687 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.838760 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.838768 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.838973 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.839019 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:02:38.839102 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.835262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:45387 (local address 127.6.241.129:44677)
0620 22:02:38.835326 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.835328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.835371 (+ 43us) server_negotiation.cc:407] Connection header received
0620 22:02:38.835470 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.835471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.835483 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.835551 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:38.835756 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.836504 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.838078 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.838682 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.838761 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.838768 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.838971 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.839014 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:02:38.839282 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.835232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:35851 (local address 127.6.241.130:47131)
0620 22:02:38.835388 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:38.835397 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:38.835464 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:38.835583 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:38.835584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:38.835586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:38.835683 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.835684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.836586 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.836586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.837985 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:38.837988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.838822 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.838823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:38.838881 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.838884 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:38.839143 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 22:02:38.839190 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":88,"thread_start_us":113,"threads_started":1}
I20260620 22:02:38.839388 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3391ddf4c36c42a4a92b728d64908552" candidate_uuid: "7fb566df7d8b466c957033cd12cd832e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e48c35195db41a7af9a551d4da954dd" is_pre_election: true
I20260620 22:02:38.839545 26401 raft_consensus.cc:2454] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb566df7d8b466c957033cd12cd832e in term 0.
I20260620 22:02:38.839861 26470 leader_election.cc:302] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb566df7d8b466c957033cd12cd832e, 9e48c35195db41a7af9a551d4da954dd; no voters:
I20260620 22:02:38.839979 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3391ddf4c36c42a4a92b728d64908552" candidate_uuid: "7fb566df7d8b466c957033cd12cd832e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e0c56fd9f34ee5941a153a13936d53" is_pre_election: true
I20260620 22:02:38.840015 26733 raft_consensus.cc:2781] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:02:38.840101 26733 raft_consensus.cc:482] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:02:38.840127 26733 raft_consensus.cc:3037] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:38.840122 26669 raft_consensus.cc:2454] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb566df7d8b466c957033cd12cd832e in term 0.
I20260620 22:02:38.841099 26733 raft_consensus.cc:504] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } }
I20260620 22:02:38.841228 26733 leader_election.cc:288] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [CANDIDATE]: Term 1 election: Requested vote from peers 9e48c35195db41a7af9a551d4da954dd (127.6.241.129:44677), d0e0c56fd9f34ee5941a153a13936d53 (127.6.241.131:35851)
I20260620 22:02:38.841362 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3391ddf4c36c42a4a92b728d64908552" candidate_uuid: "7fb566df7d8b466c957033cd12cd832e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e48c35195db41a7af9a551d4da954dd"
I20260620 22:02:38.841450 26401 raft_consensus.cc:3037] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:38.841918 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3391ddf4c36c42a4a92b728d64908552" candidate_uuid: "7fb566df7d8b466c957033cd12cd832e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e0c56fd9f34ee5941a153a13936d53"
I20260620 22:02:38.842007 26669 raft_consensus.cc:3037] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:38.842404 26401 raft_consensus.cc:2454] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb566df7d8b466c957033cd12cd832e in term 1.
I20260620 22:02:38.842569 26470 leader_election.cc:302] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb566df7d8b466c957033cd12cd832e, 9e48c35195db41a7af9a551d4da954dd; no voters:
I20260620 22:02:38.842702 26733 raft_consensus.cc:2781] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:38.842891 26733 raft_consensus.cc:686] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 1 LEADER]: Becoming Leader. State: Replica: 7fb566df7d8b466c957033cd12cd832e, State: Running, Role: LEADER
I20260620 22:02:38.842995 26733 consensus_queue.cc:237] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } }
I20260620 22:02:38.843039 26669 raft_consensus.cc:2454] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb566df7d8b466c957033cd12cd832e in term 1.
I20260620 22:02:38.843778 26255 catalog_manager.cc:5495] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb566df7d8b466c957033cd12cd832e (127.6.241.130). New cstate: current_term: 1 leader_uuid: "7fb566df7d8b466c957033cd12cd832e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb566df7d8b466c957033cd12cd832e" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 46085 } health_report { overall_health: HEALTHY } } }
I20260620 22:02:38.856266 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.854085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.6.241.130:46085)
0620 22:02:38.854139 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:38.854141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:38.854148 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:02:38.854211 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:38.854212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:38.854227 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:38.854285 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:02:38.854391 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.854921 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.855628 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:38.855850 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:38.855907 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:38.856028 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:02:38.856114 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:02:38.856184 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:38.856198 (+ 14us) server_negotiation.cc:299] Negotiation successful
0620 22:02:38.856214 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:02:38.857957 26401 raft_consensus.cc:1261] T 3391ddf4c36c42a4a92b728d64908552 P 9e48c35195db41a7af9a551d4da954dd [term 1 FOLLOWER]: Refusing update from remote peer 7fb566df7d8b466c957033cd12cd832e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:38.857988 26669 raft_consensus.cc:1261] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [term 1 FOLLOWER]: Refusing update from remote peer 7fb566df7d8b466c957033cd12cd832e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:38.858318 26733 consensus_queue.cc:1035] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e0c56fd9f34ee5941a153a13936d53" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 35851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:38.858465 26739 consensus_queue.cc:1035] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e48c35195db41a7af9a551d4da954dd" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:38.861161 26743 mvcc.cc:205] Tried to move back new op lower bound from 7299043159479472128 to 7299043159421480960. Current Snapshot: MvccSnapshot[applied={T|T < 7299043159479472128}]
I20260620 22:02:38.863291 26255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:38.863368 26255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:38.864084 26255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e9a5f00b7f146a3958039071ab7459b: Sending DeleteTablet for 3 replicas of tablet 3391ddf4c36c42a4a92b728d64908552
I20260620 22:02:38.864468 26515 tablet_service.cc:1514] Processing DeleteTablet for tablet 3391ddf4c36c42a4a92b728d64908552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:38 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:37956
I20260620 22:02:38.864511 26649 tablet_service.cc:1514] Processing DeleteTablet for tablet 3391ddf4c36c42a4a92b728d64908552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:38 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:40706
I20260620 22:02:38.864509 26381 tablet_service.cc:1514] Processing DeleteTablet for tablet 3391ddf4c36c42a4a92b728d64908552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:38 UTC) from {username='kudu', principal='kudu/127.6.241.190@KRBTEST.COM'} at 127.0.0.1:50104
I20260620 22:02:38.864701 26753 tablet_replica.cc:330] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: stopping tablet replica
I20260620 22:02:38.864799 26753 raft_consensus.cc:2229] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:02:38.864820 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 26318
I20260620 22:02:38.864892 26753 raft_consensus.cc:2258] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:38.865024 26752 tablet_replica.cc:330] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: stopping tablet replica
I20260620 22:02:38.865103 26752 raft_consensus.cc:2229] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:02:38.865218 26752 raft_consensus.cc:2258] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:38.865813 26752 ts_tablet_manager.cc:1905] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:38.866690 26753 ts_tablet_manager.cc:1905] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:38.867794 26752 ts_tablet_manager.cc:1918] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:38.867874 26752 log.cc:1196] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/3391ddf4c36c42a4a92b728d64908552
I20260620 22:02:38.868158 26752 ts_tablet_manager.cc:1939] T 3391ddf4c36c42a4a92b728d64908552 P 7fb566df7d8b466c957033cd12cd832e: Deleting consensus metadata
I20260620 22:02:38.868551 26753 ts_tablet_manager.cc:1918] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:38.868556 26240 catalog_manager.cc:4841] TS 7fb566df7d8b466c957033cd12cd832e (127.6.241.130:46085): tablet 3391ddf4c36c42a4a92b728d64908552 (table test-table [id=7a364aff3df24647a7d0e13303e22446]) successfully deleted
I20260620 22:02:38.868695 26753 log.cc:1196] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/3391ddf4c36c42a4a92b728d64908552
I20260620 22:02:38.869004 26753 ts_tablet_manager.cc:1939] T 3391ddf4c36c42a4a92b728d64908552 P d0e0c56fd9f34ee5941a153a13936d53: Deleting consensus metadata
I20260620 22:02:38.869506 26240 catalog_manager.cc:4841] TS d0e0c56fd9f34ee5941a153a13936d53 (127.6.241.131:35851): tablet 3391ddf4c36c42a4a92b728d64908552 (table test-table [id=7a364aff3df24647a7d0e13303e22446]) successfully deleted
W20260620 22:02:38.873579 26242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:44677 (error 108)
I20260620 22:02:38.873878 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 26452
I20260620 22:02:38.874490 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:38.873979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:44677 (local address 127.0.0.1:50110)
0620 22:02:38.874390 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:38.874434 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387}
W20260620 22:02:38.874722 26242 catalog_manager.cc:4584] TS 9e48c35195db41a7af9a551d4da954dd (127.6.241.129:44677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3391ddf4c36c42a4a92b728d64908552: Network error: Client connection negotiation failed: client connection to 127.6.241.129:44677: connect: Connection refused (error 111)
I20260620 22:02:38.878713 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 26586
I20260620 22:02:38.883056 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 26224
2026-06-20T22:02:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1881 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T22:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:38Z Disabled control of system clock
I20260620 22:02:38.909410 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33759
--webserver_interface=127.6.241.190
--webserver_port=0
--builtin_ntp_servers=127.6.241.148:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:33759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:02:38.987102 26759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:38.987305 26759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:38.987326 26759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:38.988408 26759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:38.988442 26759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:38.988456 26759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:38.988467 26759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:38.988479 26759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:38.989737 26759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:33759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:33759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:38.989928 26759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:38.990139 26759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:38.994706 26764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:38.994692 26767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:38.994724 26765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:38.994760 26759 server_base.cc:1034] running on GCE node
I20260620 22:02:38.995082 26759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:38.995370 26759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:38.996533 26759 hybrid_clock.cc:648] HybridClock initialized: now 1781992958996530 us; error 35 us; skew 500 ppm
I20260620 22:02:38.997671 26759 webserver.cc:460] Webserver started at http://127.6.241.190:40289/ using document root <none> and password file <none>
I20260620 22:02:38.997913 26759 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:38.997982 26759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:38.998127 26759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:38.999191 26759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "d3998cf879bb474aac738f3ccfab7307"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "4514800ea6af4bb4b3d6975884e1bb3a"
server_key_iv: "b464de38be472e92f3223b8f1e2f02c5"
server_key_version: "encryptionkey@0"
I20260620 22:02:38.999543 26759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "d3998cf879bb474aac738f3ccfab7307"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "4514800ea6af4bb4b3d6975884e1bb3a"
server_key_iv: "b464de38be472e92f3223b8f1e2f02c5"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.000770 26759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:02:39.001427 26773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.001591 26759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:39.001688 26759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "d3998cf879bb474aac738f3ccfab7307"
format_stamp: "Formatted at 2026-06-20 22:02:38 on dist-test-slave-gl2g"
server_key: "4514800ea6af4bb4b3d6975884e1bb3a"
server_key_iv: "b464de38be472e92f3223b8f1e2f02c5"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.001768 26759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:39.024434 26759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:39.024732 26759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:39.024887 26759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:39.028841 26759 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:33759
I20260620 22:02:39.028875 26825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:33759 every 8 connection(s)
I20260620 22:02:39.029211 26759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:39.030131 26826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:39.032312 26826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307: Bootstrap starting.
I20260620 22:02:39.032833 26826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:39.033063 26826 log.cc:824] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:39.033383 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 26759
I20260620 22:02:39.033497 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:39.033692 7110 external_mini_cluster.cc:1428] Setting key 6f3eaa248c85619e99fcbd72aecb9110
I20260620 22:02:39.033804 26826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307: No bootstrap required, opened a new log
I20260620 22:02:39.036360 26826 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3998cf879bb474aac738f3ccfab7307" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33759 } }
I20260620 22:02:39.036542 26826 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:39.036585 26826 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3998cf879bb474aac738f3ccfab7307, State: Initialized, Role: FOLLOWER
I20260620 22:02:39.036711 26826 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3998cf879bb474aac738f3ccfab7307" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33759 } }
I20260620 22:02:39.036788 26826 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:39.036818 26826 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:39.036855 26826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:39.037519 26826 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3998cf879bb474aac738f3ccfab7307" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33759 } }
I20260620 22:02:39.037655 26826 leader_election.cc:302] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3998cf879bb474aac738f3ccfab7307; no voters:
I20260620 22:02:39.037810 26826 leader_election.cc:288] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:39.037865 26831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:39.038012 26831 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [term 1 LEADER]: Becoming Leader. State: Replica: d3998cf879bb474aac738f3ccfab7307, State: Running, Role: LEADER
I20260620 22:02:39.038031 26826 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:39.038147 26831 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3998cf879bb474aac738f3ccfab7307" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33759 } }
I20260620 22:02:39.038551 26831 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3998cf879bb474aac738f3ccfab7307. Latest consensus state: current_term: 1 leader_uuid: "d3998cf879bb474aac738f3ccfab7307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3998cf879bb474aac738f3ccfab7307" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33759 } } }
I20260620 22:02:39.038651 26831 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:39.038844 26839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:39.039013 26832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3998cf879bb474aac738f3ccfab7307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3998cf879bb474aac738f3ccfab7307" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 33759 } } }
I20260620 22:02:39.039153 26832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:39.039340 26839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:39.040920 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.034191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.6.241.190:33759)
0620 22:02:39.034415 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.034422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.034445 (+ 23us) server_negotiation.cc:407] Connection header received
0620 22:02:39.035127 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.035135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.035191 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.035303 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:39.035562 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.036409 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.037539 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.038165 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.038628 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.039063 (+ 435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:39.039067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:39.039067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:39.039268 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:39.039307 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:39.039320 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:39.040674 (+ 1354us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.040702 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.040764 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260620 22:02:39.041963 26839 catalog_manager.cc:1347] Generated new cluster ID: edc7d4b566ba4accac47b92a0b9f9270
I20260620 22:02:39.042012 26839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:39.051515 26839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:39.052187 26839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:39.061126 26839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307: Generated new TSK 0
I20260620 22:02:39.061307 26839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:39.062295 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.129:0
--local_ip_for_outbound_sockets=127.6.241.129
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33759
--builtin_ntp_servers=127.6.241.148:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:02:39.143172 26850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:39.143396 26850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:39.143432 26850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:39.144572 26850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:39.144639 26850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:39.144680 26850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.129
I20260620 22:02:39.146014 26850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.241.129
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.129
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:39.146261 26850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:39.146585 26850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:39.151330 26855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:39.151367 26856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:39.151587 26858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:39.151857 26850 server_base.cc:1034] running on GCE node
I20260620 22:02:39.152050 26850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:39.152328 26850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:39.153483 26850 hybrid_clock.cc:648] HybridClock initialized: now 1781992959153467 us; error 32 us; skew 500 ppm
I20260620 22:02:39.154767 26850 webserver.cc:460] Webserver started at http://127.6.241.129:35197/ using document root <none> and password file <none>
I20260620 22:02:39.155014 26850 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:39.155057 26850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:39.155195 26850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:39.156210 26850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/data/instance:
uuid: "34537d8bd4ba45a5ab8c696b47432edc"
format_stamp: "Formatted at 2026-06-20 22:02:39 on dist-test-slave-gl2g"
server_key: "7774f3a40af49e0aa045dd4d3ad44635"
server_key_iv: "c566b2dd85f2a33b752f7216bf4e0fc0"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.156569 26850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance:
uuid: "34537d8bd4ba45a5ab8c696b47432edc"
format_stamp: "Formatted at 2026-06-20 22:02:39 on dist-test-slave-gl2g"
server_key: "7774f3a40af49e0aa045dd4d3ad44635"
server_key_iv: "c566b2dd85f2a33b752f7216bf4e0fc0"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.157874 26850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:39.158644 26864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.158799 26850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:39.158885 26850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/wal
uuid: "34537d8bd4ba45a5ab8c696b47432edc"
format_stamp: "Formatted at 2026-06-20 22:02:39 on dist-test-slave-gl2g"
server_key: "7774f3a40af49e0aa045dd4d3ad44635"
server_key_iv: "c566b2dd85f2a33b752f7216bf4e0fc0"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.158949 26850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:39.172883 26850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:39.173154 26850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:39.173305 26850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:39.173506 26850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:39.173836 26850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:39.173870 26850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.173892 26850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:39.173936 26850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.179739 26850 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.129:42399
I20260620 22:02:39.179827 26977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.129:42399 every 8 connection(s)
I20260620 22:02:39.180157 26850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/data/info.pb
I20260620 22:02:39.185046 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.180959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.129:38907 (local address 127.6.241.190:33759)
0620 22:02:39.181023 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.181025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.181282 (+ 257us) server_negotiation.cc:407] Connection header received
0620 22:02:39.181738 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.181739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.181753 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.181804 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:39.182414 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.182975 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.183884 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.184063 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.184108 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.184827 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:39.184828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:39.184828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:39.184833 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:39.184846 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:39.184849 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:39.184891 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.184971 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.184991 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:02:39.185230 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.180959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33759 (local address 127.6.241.129:38907)
0620 22:02:39.181169 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:39.181202 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:39.181663 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:39.181858 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:39.181861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:39.181869 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:39.182347 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.182351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.183050 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.183051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.183818 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.183820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.184129 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.184130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.184234 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.184685 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:02:39.184685 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:39.184712 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:02:39.184732 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:39.184904 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:39.184908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:39.184910 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:02:39.184960 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:02:39.185033 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
I20260620 22:02:39.185654 26978 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33759
I20260620 22:02:39.185758 26978 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:39.185928 26978 heartbeater.cc:511] Master 127.6.241.190:33759 requested a full tablet report, sending...
I20260620 22:02:39.186367 26790 ts_manager.cc:194] Registered new tserver with Master: 34537d8bd4ba45a5ab8c696b47432edc (127.6.241.129:42399)
I20260620 22:02:39.187042 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 26850
I20260620 22:02:39.187139 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-0/wal/instance
I20260620 22:02:39.187139 26790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.129:38907
I20260620 22:02:39.187345 7110 external_mini_cluster.cc:1428] Setting key 5d5ed98e20deb4208a6ff76710fe6c1f
I20260620 22:02:39.188236 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.130:0
--local_ip_for_outbound_sockets=127.6.241.130
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33759
--builtin_ntp_servers=127.6.241.148:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:02:39.266584 26981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:39.266798 26981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:39.266819 26981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:39.267891 26981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:39.267925 26981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:39.267948 26981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.130
I20260620 22:02:39.269177 26981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.241.130
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.130
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:39.269346 26981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:39.269563 26981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:39.274251 26989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:39.274338 26987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:39.274255 26986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:39.274462 26981 server_base.cc:1034] running on GCE node
I20260620 22:02:39.274705 26981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:39.274983 26981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:39.276145 26981 hybrid_clock.cc:648] HybridClock initialized: now 1781992959276119 us; error 39 us; skew 500 ppm
I20260620 22:02:39.277339 26981 webserver.cc:460] Webserver started at http://127.6.241.130:46737/ using document root <none> and password file <none>
I20260620 22:02:39.277572 26981 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:39.277640 26981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:39.277803 26981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:39.278898 26981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/data/instance:
uuid: "46456ce35cc4463ab38e81e5f5f312fa"
format_stamp: "Formatted at 2026-06-20 22:02:39 on dist-test-slave-gl2g"
server_key: "e2885e09af2298d0dbb303c71d660f09"
server_key_iv: "70226b85aa40d6e1c3d3c95627d3daf6"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.279246 26981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance:
uuid: "46456ce35cc4463ab38e81e5f5f312fa"
format_stamp: "Formatted at 2026-06-20 22:02:39 on dist-test-slave-gl2g"
server_key: "e2885e09af2298d0dbb303c71d660f09"
server_key_iv: "70226b85aa40d6e1c3d3c95627d3daf6"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.280493 26981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:39.281158 26995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.281327 26981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:39.281423 26981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/wal
uuid: "46456ce35cc4463ab38e81e5f5f312fa"
format_stamp: "Formatted at 2026-06-20 22:02:39 on dist-test-slave-gl2g"
server_key: "e2885e09af2298d0dbb303c71d660f09"
server_key_iv: "70226b85aa40d6e1c3d3c95627d3daf6"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.281502 26981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:39.325073 26981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:39.325402 26981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:39.325557 26981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:39.325803 26981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:39.326143 26981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:39.326196 26981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.326238 26981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:39.326294 26981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.331969 26981 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.130:40955
I20260620 22:02:39.332032 27108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.130:40955 every 8 connection(s)
I20260620 22:02:39.332391 26981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/data/info.pb
I20260620 22:02:39.332880 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 26981
I20260620 22:02:39.332968 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/wal/instance
I20260620 22:02:39.333184 7110 external_mini_cluster.cc:1428] Setting key c8a274238508b2faf19929ed374c2523
I20260620 22:02:39.334198 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.241.131:0
--local_ip_for_outbound_sockets=127.6.241.131
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33759
--builtin_ntp_servers=127.6.241.148:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:02:39.338558 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.333155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:57213 (local address 127.6.241.190:33759)
0620 22:02:39.333221 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.333223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.333537 (+ 314us) server_negotiation.cc:407] Connection header received
0620 22:02:39.334171 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.334171 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.334182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.334224 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:39.335150 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.335814 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.336800 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.337137 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.337222 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.338235 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:39.338237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:39.338237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:39.338243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:39.338261 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:39.338266 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:39.338329 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.338387 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.338413 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:02:39.338814 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.333158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33759 (local address 127.6.241.130:57213)
0620 22:02:39.333375 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:39.333420 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:39.334074 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:39.334323 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:39.334327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:39.334339 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:39.335052 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.335059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.335906 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.335906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.336697 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.336700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.337242 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.337243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.337360 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.338098 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:02:39.338099 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:39.338127 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:02:39.338149 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:39.338329 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:39.338332 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:39.338334 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:02:39.338416 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:02:39.338536 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260620 22:02:39.339298 27109 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33759
I20260620 22:02:39.339423 27109 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:39.339627 27109 heartbeater.cc:511] Master 127.6.241.190:33759 requested a full tablet report, sending...
I20260620 22:02:39.340030 26790 ts_manager.cc:194] Registered new tserver with Master: 46456ce35cc4463ab38e81e5f5f312fa (127.6.241.130:40955)
I20260620 22:02:39.340422 26790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.130:57213
W20260620 22:02:39.413935 27112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:39.414153 27112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:39.414184 27112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:39.415279 27112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:39.415336 27112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:39.415374 27112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.241.131
I20260620 22:02:39.416684 27112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.241.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.241.131
--webserver_port=0
--tserver_master_addrs=127.6.241.190:33759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.241.131
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:39.416931 27112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:39.417223 27112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:39.421984 27120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:39.422014 27118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:39.421960 27117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:39.422134 27112 server_base.cc:1034] running on GCE node
I20260620 22:02:39.422322 27112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:39.422569 27112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:39.423745 27112 hybrid_clock.cc:648] HybridClock initialized: now 1781992959423735 us; error 30 us; skew 500 ppm
I20260620 22:02:39.424894 27112 webserver.cc:460] Webserver started at http://127.6.241.131:38701/ using document root <none> and password file <none>
I20260620 22:02:39.425137 27112 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:39.425225 27112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:39.425400 27112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:39.426788 27112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/data/instance:
uuid: "8276dda502f44382909ef596de87156d"
format_stamp: "Formatted at 2026-06-20 22:02:39 on dist-test-slave-gl2g"
server_key: "3dd1d8dcb6d8ca0757f80ab9619f64c6"
server_key_iv: "c90ea87b16d0a0c093e242c5a07a1025"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.427132 27112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance:
uuid: "8276dda502f44382909ef596de87156d"
format_stamp: "Formatted at 2026-06-20 22:02:39 on dist-test-slave-gl2g"
server_key: "3dd1d8dcb6d8ca0757f80ab9619f64c6"
server_key_iv: "c90ea87b16d0a0c093e242c5a07a1025"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.428380 27112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:39.429114 27126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.429271 27112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:39.429373 27112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/wal
uuid: "8276dda502f44382909ef596de87156d"
format_stamp: "Formatted at 2026-06-20 22:02:39 on dist-test-slave-gl2g"
server_key: "3dd1d8dcb6d8ca0757f80ab9619f64c6"
server_key_iv: "c90ea87b16d0a0c093e242c5a07a1025"
server_key_version: "encryptionkey@0"
I20260620 22:02:39.429464 27112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:39.443749 27112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:39.443976 27112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:39.444072 27112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:39.444217 27112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:02:39.444528 27112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:02:39.444595 27112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.444634 27112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:02:39.444664 27112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:39.450134 27112 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.131:46033
I20260620 22:02:39.450201 27239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.131:46033 every 8 connection(s)
I20260620 22:02:39.450513 27112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/data/info.pb
I20260620 22:02:39.455284 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.451210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.131:59839 (local address 127.6.241.190:33759)
0620 22:02:39.451295 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.451297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.451528 (+ 231us) server_negotiation.cc:407] Connection header received
0620 22:02:39.451999 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.452000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.452008 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.452057 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:39.452666 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.453134 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.454038 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.454213 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.454267 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.455041 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:39.455042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:39.455042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:39.455046 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:39.455059 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:39.455063 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:39.455103 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.455203 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.455223 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 22:02:39.455456 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.451203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:33759 (local address 127.6.241.131:59839)
0620 22:02:39.451387 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:39.451435 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:39.451890 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:39.452113 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:39.452116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:39.452124 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:02:39.452596 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.452600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.453211 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.453211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.453961 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.453963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.454286 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.454287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.454390 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.454902 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:02:39.454902 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:02:39.454928 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:02:39.454949 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:02:39.455133 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:02:39.455137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:02:39.455139 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:02:39.455189 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:02:39.455262 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260620 22:02:39.455942 27240 heartbeater.cc:347] Connected to a master server at 127.6.241.190:33759
I20260620 22:02:39.456014 27240 heartbeater.cc:464] Registering TS with master...
I20260620 22:02:39.456141 27240 heartbeater.cc:511] Master 127.6.241.190:33759 requested a full tablet report, sending...
I20260620 22:02:39.456462 26790 ts_manager.cc:194] Registered new tserver with Master: 8276dda502f44382909ef596de87156d (127.6.241.131:46033)
I20260620 22:02:39.456774 26790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.241.131:59839
I20260620 22:02:39.459115 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 27112
I20260620 22:02:39.459223 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/wal/instance
I20260620 22:02:39.459408 7110 external_mini_cluster.cc:1428] Setting key 17fbf2f69cf2e02d7dd220934bb54eec
I20260620 22:02:39.460302 7110 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:02:39.464047 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.461522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.6.241.190:33759)
0620 22:02:39.461578 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.461580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.461724 (+ 144us) server_negotiation.cc:407] Connection header received
0620 22:02:39.461785 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.461785 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.461800 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.461846 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:39.461996 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.462424 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.463168 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.463332 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.463382 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.463871 (+ 489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:39.463872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:39.463873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:39.463876 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:39.463885 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:39.463889 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:39.463931 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.463987 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.464001 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:02:39.465391 26790 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:02:39.466928 26790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:02:39.473173 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.469025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.6.241.131:46033)
0620 22:02:39.469399 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.469402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.469410 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:02:39.469435 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.469436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.469447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.469507 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:39.469686 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.470940 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.472157 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.472809 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.472888 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.472974 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.473068 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.473104 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":82,"threads_started":1}
I20260620 22:02:39.473194 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.468869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:46033 (local address 127.0.0.1:51100)
0620 22:02:39.468992 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:39.469030 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:39.469113 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:39.469533 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:39.469536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:39.469538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:39.469602 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.469604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.471024 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.471025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.472079 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.472081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.472897 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.472899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.472975 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.472990 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:02:39.473058 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:02:39.473117 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":61,"thread_start_us":75,"threads_started":1}
I20260620 22:02:39.473455 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.469086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.130:40955 (local address 127.0.0.1:32832)
0620 22:02:39.469565 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:39.469597 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:39.469642 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:39.469856 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:39.469857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:39.469859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:39.469889 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.469890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.470756 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.470757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.471952 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.471955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.472899 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.472900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.472975 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.472990 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:02:39.473058 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:02:39.473117 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260620 22:02:39.473678 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.469197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.6.241.130:40955)
0620 22:02:39.469371 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.469377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.469705 (+ 328us) server_negotiation.cc:407] Connection header received
0620 22:02:39.469748 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.469750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.469771 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.469859 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:39.469946 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.470670 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.472042 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.472810 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.472888 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.472974 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.473165 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.473209 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260620 22:02:39.473925 27174 tablet_service.cc:1467] Processing CreateTablet for tablet a8e98eb29a7b47a7b178e14b30b1337a (DEFAULT_TABLE table=test-table [id=6abf94ca59d04113a827ba4ace97faea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:39.474148 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.468913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42399 (local address 127.0.0.1:40264)
0620 22:02:39.469197 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:39.469208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:39.469271 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:39.469619 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:39.469620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:39.469622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:39.469670 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.469672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.471063 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.471064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.471962 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.471965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.473931 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.473932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.473992 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.473997 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:02:39.474039 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:02:39.474080 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":30,"thread_start_us":86,"threads_started":1}
I20260620 22:02:39.474287 27174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e98eb29a7b47a7b178e14b30b1337a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:39.474328 27043 tablet_service.cc:1467] Processing CreateTablet for tablet a8e98eb29a7b47a7b178e14b30b1337a (DEFAULT_TABLE table=test-table [id=6abf94ca59d04113a827ba4ace97faea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:39.474465 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.469193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.6.241.129:42399)
0620 22:02:39.469433 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.469437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.469452 (+ 15us) server_negotiation.cc:407] Connection header received
0620 22:02:39.469484 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.469503 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.469520 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.469605 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:39.469946 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.471001 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.473010 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.473716 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.473795 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.474313 (+ 518us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.474341 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.474376 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260620 22:02:39.474680 27043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e98eb29a7b47a7b178e14b30b1337a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:39.475104 26912 tablet_service.cc:1467] Processing CreateTablet for tablet a8e98eb29a7b47a7b178e14b30b1337a (DEFAULT_TABLE table=test-table [id=6abf94ca59d04113a827ba4ace97faea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:02:39.475430 26912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e98eb29a7b47a7b178e14b30b1337a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:39.477072 27254 tablet_bootstrap.cc:492] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: Bootstrap starting.
I20260620 22:02:39.477553 27254 tablet_bootstrap.cc:654] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:39.477823 27254 log.cc:824] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:39.478014 27255 tablet_bootstrap.cc:492] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: Bootstrap starting.
I20260620 22:02:39.478559 27254 tablet_bootstrap.cc:492] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: No bootstrap required, opened a new log
I20260620 22:02:39.478605 27255 tablet_bootstrap.cc:654] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:39.478646 27254 ts_tablet_manager.cc:1397] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:02:39.478720 27257 tablet_bootstrap.cc:492] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc: Bootstrap starting.
I20260620 22:02:39.478952 27255 log.cc:824] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:39.479326 27257 tablet_bootstrap.cc:654] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:39.479600 27257 log.cc:824] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:39.479836 27255 tablet_bootstrap.cc:492] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: No bootstrap required, opened a new log
I20260620 22:02:39.479904 27255 ts_tablet_manager.cc:1397] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:02:39.480304 27257 tablet_bootstrap.cc:492] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc: No bootstrap required, opened a new log
I20260620 22:02:39.480376 27257 ts_tablet_manager.cc:1397] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:02:39.481381 27254 raft_consensus.cc:348] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } }
I20260620 22:02:39.481549 27254 raft_consensus.cc:374] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:39.481585 27254 raft_consensus.cc:729] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8276dda502f44382909ef596de87156d, State: Initialized, Role: FOLLOWER
I20260620 22:02:39.481719 27254 consensus_queue.cc:260] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } }
I20260620 22:02:39.481958 27254 ts_tablet_manager.cc:1428] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:02:39.482076 27240 heartbeater.cc:503] Master 127.6.241.190:33759 was elected leader, sending a full tablet report...
I20260620 22:02:39.482092 27255 raft_consensus.cc:348] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } }
I20260620 22:02:39.482237 27255 raft_consensus.cc:374] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:39.482276 27255 raft_consensus.cc:729] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46456ce35cc4463ab38e81e5f5f312fa, State: Initialized, Role: FOLLOWER
I20260620 22:02:39.482406 27255 consensus_queue.cc:260] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } }
I20260620 22:02:39.482700 27255 ts_tablet_manager.cc:1428] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:02:39.482760 27109 heartbeater.cc:503] Master 127.6.241.190:33759 was elected leader, sending a full tablet report...
I20260620 22:02:39.482981 27257 raft_consensus.cc:348] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } }
I20260620 22:02:39.483148 27257 raft_consensus.cc:374] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:39.483192 27257 raft_consensus.cc:729] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34537d8bd4ba45a5ab8c696b47432edc, State: Initialized, Role: FOLLOWER
I20260620 22:02:39.483300 27257 consensus_queue.cc:260] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } }
I20260620 22:02:39.483542 27257 ts_tablet_manager.cc:1428] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:02:39.483593 26978 heartbeater.cc:503] Master 127.6.241.190:33759 was elected leader, sending a full tablet report...
W20260620 22:02:39.583542 27110 tablet.cc:2369] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:02:39.639115 27261 raft_consensus.cc:482] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:02:39.639246 27261 raft_consensus.cc:504] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } }
I20260620 22:02:39.639622 27261 leader_election.cc:288] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8276dda502f44382909ef596de87156d (127.6.241.131:46033), 34537d8bd4ba45a5ab8c696b47432edc (127.6.241.129:42399)
I20260620 22:02:39.643736 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.639941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:57195 (local address 127.6.241.131:46033)
0620 22:02:39.639990 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.639993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.640006 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:02:39.640071 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.640072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.640082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.640157 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:39.640387 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.641195 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.642788 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.643349 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.643432 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.643441 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.643614 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.643669 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:02:39.643736 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.639959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.241.130:54245 (local address 127.6.241.129:42399)
0620 22:02:39.640036 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.640039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.640231 (+ 192us) server_negotiation.cc:407] Connection header received
0620 22:02:39.640259 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.640260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.640270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.640314 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:02:39.640489 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.641191 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.642789 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.643349 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.643439 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.643447 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.643614 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.643659 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 22:02:39.643761 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.639855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.131:46033 (local address 127.6.241.130:57195)
0620 22:02:39.639932 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:39.639946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:39.639994 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:39.640190 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:39.640192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:39.640193 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:39.640272 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.640274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.641309 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.641310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.642678 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.642682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.643447 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.643448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.643528 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.643532 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:02:39.643590 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 22:02:39.643649 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":90}
I20260620 22:02:39.643766 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.639855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42399 (local address 127.6.241.130:54245)
0620 22:02:39.640126 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:39.640134 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:02:39.640210 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:02:39.640343 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:02:39.640344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:02:39.640345 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:02:39.640405 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.640406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.641295 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.641296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.642678 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:02:39.642682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.643448 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.643448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:02:39.643533 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.643536 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:02:39.643590 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 22:02:39.643649 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":130,"thread_start_us":106,"threads_started":1}
I20260620 22:02:39.644245 26932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e98eb29a7b47a7b178e14b30b1337a" candidate_uuid: "46456ce35cc4463ab38e81e5f5f312fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34537d8bd4ba45a5ab8c696b47432edc" is_pre_election: true
I20260620 22:02:39.644284 27194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e98eb29a7b47a7b178e14b30b1337a" candidate_uuid: "46456ce35cc4463ab38e81e5f5f312fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8276dda502f44382909ef596de87156d" is_pre_election: true
I20260620 22:02:39.644449 27194 raft_consensus.cc:2454] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46456ce35cc4463ab38e81e5f5f312fa in term 0.
I20260620 22:02:39.644450 26932 raft_consensus.cc:2454] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46456ce35cc4463ab38e81e5f5f312fa in term 0.
I20260620 22:02:39.644708 26999 leader_election.cc:302] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34537d8bd4ba45a5ab8c696b47432edc, 46456ce35cc4463ab38e81e5f5f312fa; no voters:
I20260620 22:02:39.644896 27261 raft_consensus.cc:2781] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:02:39.644964 27261 raft_consensus.cc:482] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:02:39.644989 27261 raft_consensus.cc:3037] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:39.645714 27261 raft_consensus.cc:504] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } }
I20260620 22:02:39.645874 27261 leader_election.cc:288] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [CANDIDATE]: Term 1 election: Requested vote from peers 8276dda502f44382909ef596de87156d (127.6.241.131:46033), 34537d8bd4ba45a5ab8c696b47432edc (127.6.241.129:42399)
I20260620 22:02:39.646034 26932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e98eb29a7b47a7b178e14b30b1337a" candidate_uuid: "46456ce35cc4463ab38e81e5f5f312fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34537d8bd4ba45a5ab8c696b47432edc"
I20260620 22:02:39.646052 27194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e98eb29a7b47a7b178e14b30b1337a" candidate_uuid: "46456ce35cc4463ab38e81e5f5f312fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8276dda502f44382909ef596de87156d"
I20260620 22:02:39.646136 26932 raft_consensus.cc:3037] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:39.646126 27194 raft_consensus.cc:3037] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:39.646965 27194 raft_consensus.cc:2454] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46456ce35cc4463ab38e81e5f5f312fa in term 1.
I20260620 22:02:39.647089 26932 raft_consensus.cc:2454] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46456ce35cc4463ab38e81e5f5f312fa in term 1.
I20260620 22:02:39.647140 26998 leader_election.cc:302] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46456ce35cc4463ab38e81e5f5f312fa, 8276dda502f44382909ef596de87156d; no voters:
I20260620 22:02:39.647295 27261 raft_consensus.cc:2781] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:39.647424 27261 raft_consensus.cc:686] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 1 LEADER]: Becoming Leader. State: Replica: 46456ce35cc4463ab38e81e5f5f312fa, State: Running, Role: LEADER
I20260620 22:02:39.647526 27261 consensus_queue.cc:237] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } }
I20260620 22:02:39.648439 26789 catalog_manager.cc:5495] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 46456ce35cc4463ab38e81e5f5f312fa (127.6.241.130). New cstate: current_term: 1 leader_uuid: "46456ce35cc4463ab38e81e5f5f312fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46456ce35cc4463ab38e81e5f5f312fa" member_type: VOTER last_known_addr { host: "127.6.241.130" port: 40955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 } health_report { overall_health: UNKNOWN } } }
W20260620 22:02:39.681283 26979 tablet.cc:2369] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:02:39.697454 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.694988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32844 (local address 127.6.241.130:40955)
0620 22:02:39.695054 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:39.695056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:39.695064 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:02:39.695109 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:39.695110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:39.695130 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:39.695211 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:02:39.695360 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.695831 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.696614 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:39.696811 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:39.696890 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:39.697065 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:02:39.697186 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:02:39.697286 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:39.697319 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 22:02:39.697339 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:02:39.699276 26932 raft_consensus.cc:1261] T a8e98eb29a7b47a7b178e14b30b1337a P 34537d8bd4ba45a5ab8c696b47432edc [term 1 FOLLOWER]: Refusing update from remote peer 46456ce35cc4463ab38e81e5f5f312fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:39.699378 27194 raft_consensus.cc:1261] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [term 1 FOLLOWER]: Refusing update from remote peer 46456ce35cc4463ab38e81e5f5f312fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:02:39.699656 27264 consensus_queue.cc:1035] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "34537d8bd4ba45a5ab8c696b47432edc" member_type: VOTER last_known_addr { host: "127.6.241.129" port: 42399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:02:39.699801 27261 consensus_queue.cc:1035] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8276dda502f44382909ef596de87156d" member_type: VOTER last_known_addr { host: "127.6.241.131" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:02:39.701179 27241 tablet.cc:2369] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:02:39.701998 27267 mvcc.cc:205] Tried to move back new op lower bound from 7299043162925457408 to 7299043162716827648. Current Snapshot: MvccSnapshot[applied={T|T < 7299043162925457408}]
I20260620 22:02:39.702195 27268 mvcc.cc:205] Tried to move back new op lower bound from 7299043162925457408 to 7299043162716827648. Current Snapshot: MvccSnapshot[applied={T|T < 7299043162925457408}]
I20260620 22:02:39.704494 26789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:39.704581 26789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:02:39.705294 26789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3998cf879bb474aac738f3ccfab7307: Sending DeleteTablet for 3 replicas of tablet a8e98eb29a7b47a7b178e14b30b1337a
I20260620 22:02:39.705670 27174 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e98eb29a7b47a7b178e14b30b1337a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:39 UTC) from {username='slave'} at 127.0.0.1:51100
I20260620 22:02:39.705668 26912 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e98eb29a7b47a7b178e14b30b1337a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:39 UTC) from {username='slave'} at 127.0.0.1:40264
I20260620 22:02:39.705866 27276 tablet_replica.cc:330] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: stopping tablet replica
I20260620 22:02:39.705878 27043 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e98eb29a7b47a7b178e14b30b1337a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:02:39 UTC) from {username='slave'} at 127.0.0.1:32832
I20260620 22:02:39.705964 27276 raft_consensus.cc:2229] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:02:39.705958 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 26850
I20260620 22:02:39.707230 27276 raft_consensus.cc:2258] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:39.707702 27276 ts_tablet_manager.cc:1905] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:39.708213 27278 tablet_replica.cc:330] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: stopping tablet replica
I20260620 22:02:39.708352 27278 raft_consensus.cc:2229] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:02:39.708501 27278 raft_consensus.cc:2258] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:02:39.708993 27278 ts_tablet_manager.cc:1905] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:02:39.709354 27276 ts_tablet_manager.cc:1918] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:39.709455 27276 log.cc:1196] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-2/wal/wals/a8e98eb29a7b47a7b178e14b30b1337a
I20260620 22:02:39.709720 27276 ts_tablet_manager.cc:1939] T a8e98eb29a7b47a7b178e14b30b1337a P 8276dda502f44382909ef596de87156d: Deleting consensus metadata
I20260620 22:02:39.710122 26776 catalog_manager.cc:4841] TS 8276dda502f44382909ef596de87156d (127.6.241.131:46033): tablet a8e98eb29a7b47a7b178e14b30b1337a (table test-table [id=6abf94ca59d04113a827ba4ace97faea]) successfully deleted
I20260620 22:02:39.710916 27278 ts_tablet_manager.cc:1918] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:02:39.711051 27278 log.cc:1196] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: Deleting WAL directory at /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992718478901-7110-0/minicluster-data/ts-1/wal/wals/a8e98eb29a7b47a7b178e14b30b1337a
I20260620 22:02:39.711360 27278 ts_tablet_manager.cc:1939] T a8e98eb29a7b47a7b178e14b30b1337a P 46456ce35cc4463ab38e81e5f5f312fa: Deleting consensus metadata
I20260620 22:02:39.711777 26776 catalog_manager.cc:4841] TS 46456ce35cc4463ab38e81e5f5f312fa (127.6.241.130:40955): tablet a8e98eb29a7b47a7b178e14b30b1337a (table test-table [id=6abf94ca59d04113a827ba4ace97faea]) successfully deleted
W20260620 22:02:39.712430 26777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.241.129:42399 (error 108)
I20260620 22:02:39.712913 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:39.712776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.129:42399 (local address 127.0.0.1:40276)
0620 22:02:39.712837 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:02:39.712880 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.129:42399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 22:02:39.712949 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 26981
W20260620 22:02:39.713909 26777 catalog_manager.cc:4584] TS 34537d8bd4ba45a5ab8c696b47432edc (127.6.241.129:42399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8e98eb29a7b47a7b178e14b30b1337a: Network error: Client connection negotiation failed: client connection to 127.6.241.129:42399: connect: Connection refused (error 111)
I20260620 22:02:39.717837 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 27112
I20260620 22:02:39.722785 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 26759
2026-06-20T22:02:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (839 ms)
[----------] 6 tests from EncryptionPolicyTest (8195 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:39 dist-test-slave-gl2g krb5kdc[27282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:39 dist-test-slave-gl2g krb5kdc[27282](info): set up 2 sockets
Jun 20 22:02:39 dist-test-slave-gl2g krb5kdc[27282](info): commencing operation
krb5kdc: starting...
W20260620 22:02:40.754808 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:40 dist-test-slave-gl2g krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:40.873620 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:40.954191 27295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:40.954437 27295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:40.954475 27295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:40.955668 27295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:40.955744 27295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:40.955771 27295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:40.955799 27295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:40.955826 27295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:40.957271 27295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:40.957533 27295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:40.957757 27295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:40.962520 27301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:40.962520 27300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:40.962589 27295 server_base.cc:1034] running on GCE node
W20260620 22:02:40.962508 27303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:40.962935 27295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:40.963227 27295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:40.964398 27295 hybrid_clock.cc:648] HybridClock initialized: now 1781992960964396 us; error 40 us; skew 500 ppm
Jun 20 22:02:40 dist-test-slave-gl2g krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:40.966850 27295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:40.967291 27295 webserver.cc:460] Webserver started at http://127.0.0.1:37211/ using document root <none> and password file <none>
I20260620 22:02:40.967473 27295 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:40.967511 27295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:40.967598 27295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:40.968582 27295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "19260bf609f04e8d827b1ad8f9692a59"
format_stamp: "Formatted at 2026-06-20 22:02:40 on dist-test-slave-gl2g"
server_key: "dab5e25c1e62bd61d18337c80352b9c3"
server_key_iv: "88d1d68989f0e2bad7cb1881e5359d6a"
server_key_version: "encryptionkey@0"
I20260620 22:02:40.968894 27295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "19260bf609f04e8d827b1ad8f9692a59"
format_stamp: "Formatted at 2026-06-20 22:02:40 on dist-test-slave-gl2g"
server_key: "dab5e25c1e62bd61d18337c80352b9c3"
server_key_iv: "88d1d68989f0e2bad7cb1881e5359d6a"
server_key_version: "encryptionkey@0"
I20260620 22:02:40.970120 27295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:40.970839 27310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:40.970996 27295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:40.971087 27295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "19260bf609f04e8d827b1ad8f9692a59"
format_stamp: "Formatted at 2026-06-20 22:02:40 on dist-test-slave-gl2g"
server_key: "dab5e25c1e62bd61d18337c80352b9c3"
server_key_iv: "88d1d68989f0e2bad7cb1881e5359d6a"
server_key_version: "encryptionkey@0"
I20260620 22:02:40.971140 27295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:41.001230 27295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:41.003430 27295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:41.003548 27295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:41.007508 27295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46621
I20260620 22:02:41.007587 27362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46621 every 8 connection(s)
I20260620 22:02:41.007941 27295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:41.008606 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 27295
I20260620 22:02:41.008725 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:41.008927 7110 external_mini_cluster.cc:1428] Setting key f09fc8763448974bfba91de2297893e9
I20260620 22:02:41.009019 27363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:41.012351 27363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59: Bootstrap starting.
I20260620 22:02:41.013051 27363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:41.013340 27363 log.cc:824] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:41.014047 27363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59: No bootstrap required, opened a new log
Jun 20 22:02:41 dist-test-slave-gl2g krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:02:41.016549 27363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19260bf609f04e8d827b1ad8f9692a59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } }
I20260620 22:02:41.016732 27363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:41.016773 27363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19260bf609f04e8d827b1ad8f9692a59, State: Initialized, Role: FOLLOWER
I20260620 22:02:41.016950 27363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19260bf609f04e8d827b1ad8f9692a59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } }
I20260620 22:02:41.017026 27363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:41.017048 27363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:41.017114 27363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:41.017635 27363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19260bf609f04e8d827b1ad8f9692a59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } }
I20260620 22:02:41.017719 27363 leader_election.cc:302] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19260bf609f04e8d827b1ad8f9692a59; no voters:
I20260620 22:02:41.017948 27363 leader_election.cc:288] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:41.017979 27368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:41.018086 27368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [term 1 LEADER]: Becoming Leader. State: Replica: 19260bf609f04e8d827b1ad8f9692a59, State: Running, Role: LEADER
I20260620 22:02:41.018173 27368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19260bf609f04e8d827b1ad8f9692a59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } }
I20260620 22:02:41.018210 27363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:41.018491 27368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19260bf609f04e8d827b1ad8f9692a59. Latest consensus state: current_term: 1 leader_uuid: "19260bf609f04e8d827b1ad8f9692a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19260bf609f04e8d827b1ad8f9692a59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } } }
I20260620 22:02:41.018579 27368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:41.018500 27370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19260bf609f04e8d827b1ad8f9692a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19260bf609f04e8d827b1ad8f9692a59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } } }
I20260620 22:02:41.018811 27370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:41.018929 27374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:41.019474 27374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:41.020449 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:41.009503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59959 (local address 127.0.0.1:46621)
0620 22:02:41.009693 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:41.009700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:41.009726 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:02:41.010130 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:41.010137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:41.010183 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:41.010296 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:41.011097 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:41.011985 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:41.013022 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:41.013512 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:41.013629 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:41.016281 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:41.016287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:41.016288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:41.016302 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:41.018451 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:41.018971 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:41.018974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:41.018974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:41.019098 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:41.019335 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:41.019336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:41.019336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:41.019665 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:41.019828 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:41.020141 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 22:02:41.020243 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260620 22:02:41.021324 27374 catalog_manager.cc:1347] Generated new cluster ID: 4c450019f11e4c2fbe31d3a346ecbd44
I20260620 22:02:41.021458 27374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:41.021549 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:41.029222 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:41.022866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34539 (local address 127.0.0.1:46621)
0620 22:02:41.022900 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:41.022902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:41.023026 (+ 124us) server_negotiation.cc:407] Connection header received
0620 22:02:41.023096 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:41.023097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:41.023107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:41.023161 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:41.023667 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:41.024156 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:41.024854 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:41.025082 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:41.025161 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:41.026363 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:41.026365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:41.026365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:41.026372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:41.027869 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:41.028323 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:41.028324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:41.028324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:41.028380 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:41.028612 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:41.028613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:41.028613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:41.028729 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:41.028837 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:41.028972 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 22:02:41.029041 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:02:41.030419 27374 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:41.031066 27374 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:41.037075 27374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19260bf609f04e8d827b1ad8f9692a59: Generated new TSK 0
I20260620 22:02:41.037290 27374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:41.042891 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 27295
W20260620 22:02:41.047158 27280 connection.cc:535] client connection to 127.0.0.1:46621 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46621): Connection reset by peer (error 104)
2026-06-20T22:02:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:41 dist-test-slave-gl2g krb5kdc[27394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:41 dist-test-slave-gl2g krb5kdc[27394](info): set up 2 sockets
Jun 20 22:02:41 dist-test-slave-gl2g krb5kdc[27394](info): commencing operation
krb5kdc: starting...
W20260620 22:02:42.074546 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:42 dist-test-slave-gl2g krb5kdc[27394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:42.197649 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:42.279237 27407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:42.279510 27407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:42.279551 27407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:42.280788 27407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:42.280876 27407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:42.280901 27407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:42.280931 27407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:42.280956 27407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:42.282646 27407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42567
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:42.282938 27407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:42.283182 27407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:42.288264 27412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:42.288255 27415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:42.288234 27413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:42.288475 27407 server_base.cc:1034] running on GCE node
I20260620 22:02:42.288717 27407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:42.289072 27407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:42.290241 27407 hybrid_clock.cc:648] HybridClock initialized: now 1781992962290225 us; error 48 us; skew 500 ppm
Jun 20 22:02:42 dist-test-slave-gl2g krb5kdc[27394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:42.297631 27407 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:42.298158 27407 webserver.cc:460] Webserver started at http://127.0.0.1:40687/ using document root <none> and password file <none>
I20260620 22:02:42.298415 27407 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:42.298487 27407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:42.298681 27407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:42.299764 27407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c"
format_stamp: "Formatted at 2026-06-20 22:02:42 on dist-test-slave-gl2g"
server_key: "4576b51b74290dd5b4f297d7c9bf68bf"
server_key_iv: "2e702baf64a68daecd1add3b986e7de6"
server_key_version: "encryptionkey@0"
I20260620 22:02:42.300135 27407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c"
format_stamp: "Formatted at 2026-06-20 22:02:42 on dist-test-slave-gl2g"
server_key: "4576b51b74290dd5b4f297d7c9bf68bf"
server_key_iv: "2e702baf64a68daecd1add3b986e7de6"
server_key_version: "encryptionkey@0"
I20260620 22:02:42.301493 27407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:42.302259 27422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:42.302456 27407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:42.302556 27407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c"
format_stamp: "Formatted at 2026-06-20 22:02:42 on dist-test-slave-gl2g"
server_key: "4576b51b74290dd5b4f297d7c9bf68bf"
server_key_iv: "2e702baf64a68daecd1add3b986e7de6"
server_key_version: "encryptionkey@0"
I20260620 22:02:42.302660 27407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:42.332978 27407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:42.334968 27407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:42.335147 27407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:42.339157 27407 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42567
I20260620 22:02:42.339215 27474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42567 every 8 connection(s)
I20260620 22:02:42.339591 27407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:42.340582 27475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:42.342974 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 27407
I20260620 22:02:42.343096 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:42.343107 27475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c: Bootstrap starting.
I20260620 22:02:42.343283 7110 external_mini_cluster.cc:1428] Setting key 6f5c9f315e0327ff9ed8bdfde3954295
I20260620 22:02:42.343840 27475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:42.344208 27475 log.cc:824] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:42.344928 27475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c: No bootstrap required, opened a new log
I20260620 22:02:42.347077 27475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42567 } }
I20260620 22:02:42.347251 27475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:42.347287 27475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e25aa6bf8fb4496b5b97e0ec7be7c9c, State: Initialized, Role: FOLLOWER
I20260620 22:02:42.347435 27475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42567 } }
I20260620 22:02:42.347505 27475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:42.347527 27475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:42.347599 27475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:42.348276 27475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42567 } }
I20260620 22:02:42.348409 27475 leader_election.cc:302] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e25aa6bf8fb4496b5b97e0ec7be7c9c; no voters:
I20260620 22:02:42.348592 27475 leader_election.cc:288] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:42.348680 27480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:42.348850 27480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [term 1 LEADER]: Becoming Leader. State: Replica: 1e25aa6bf8fb4496b5b97e0ec7be7c9c, State: Running, Role: LEADER
I20260620 22:02:42.348862 27475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:42.348973 27480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42567 } }
I20260620 22:02:42.349655 27478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:42.343873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49725 (local address 127.0.0.1:42567)
0620 22:02:42.344081 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:42.344089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:42.344188 (+ 99us) server_negotiation.cc:407] Connection header received
0620 22:02:42.344828 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:42.344837 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:42.344888 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:42.345015 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:42.345304 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:42.346195 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:42.347367 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:42.347924 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:42.348006 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:42.348861 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:42.348865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:42.348866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:42.349076 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:42.349116 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:42.349131 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:42.349223 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:42.349368 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 22:02:42.349399 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260620 22:02:42.349725 27482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e25aa6bf8fb4496b5b97e0ec7be7c9c. Latest consensus state: current_term: 1 leader_uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42567 } } }
I20260620 22:02:42.349807 27482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:42.350086 27481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e25aa6bf8fb4496b5b97e0ec7be7c9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42567 } } }
I20260620 22:02:42.350162 27481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:42.350315 27489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:42.350569 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:42.350828 27489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:42.352499 27489 catalog_manager.cc:1347] Generated new cluster ID: 68d4409a3aec413c833fd56181878ab0
I20260620 22:02:42.352555 27489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:42.355774 27478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:42.352153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44007 (local address 127.0.0.1:42567)
0620 22:02:42.352186 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:42.352188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:42.352335 (+ 147us) server_negotiation.cc:407] Connection header received
0620 22:02:42.352382 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:42.352383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:42.352393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:42.352445 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:42.352608 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:42.353340 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:42.354357 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:42.354727 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:42.354809 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:42.355521 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:42.355522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:42.355523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:42.355527 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:42.355544 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:42.355549 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:42.355617 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:42.355688 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 22:02:42.355699 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:02:42.362190 27489 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:42.362977 27489 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:42.374061 27489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e25aa6bf8fb4496b5b97e0ec7be7c9c: Generated new TSK 0
I20260620 22:02:42.374353 27489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:42.377179 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 27407
W20260620 22:02:42.381294 27392 connection.cc:535] client connection to 127.0.0.1:42567 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42567): Connection reset by peer (error 104)
2026-06-20T22:02:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:42 dist-test-slave-gl2g krb5kdc[27506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:42 dist-test-slave-gl2g krb5kdc[27506](info): set up 2 sockets
Jun 20 22:02:42 dist-test-slave-gl2g krb5kdc[27506](info): commencing operation
krb5kdc: starting...
W20260620 22:02:43.434773 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:43 dist-test-slave-gl2g krb5kdc[27506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:43.557606 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:43.639147 27519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:43.639406 27519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:43.639446 27519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:43.640614 27519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:43.640686 27519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:43.640712 27519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:43.640744 27519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:43.640769 27519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:43.642273 27519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:43.642556 27519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:43.642814 27519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:43.647754 27524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:43.647747 27527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:43.647747 27525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:43.647996 27519 server_base.cc:1034] running on GCE node
I20260620 22:02:43.648253 27519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:43.648550 27519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:43.649729 27519 hybrid_clock.cc:648] HybridClock initialized: now 1781992963649722 us; error 35 us; skew 500 ppm
Jun 20 22:02:43 dist-test-slave-gl2g krb5kdc[27506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:43.652204 27519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:43.652657 27519 webserver.cc:460] Webserver started at http://127.0.0.1:38947/ using document root <none> and password file <none>
I20260620 22:02:43.652896 27519 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:43.652966 27519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:43.653120 27519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:43.654201 27519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "72c3c86f4fc24c89a40c042dd568e9b0"
format_stamp: "Formatted at 2026-06-20 22:02:43 on dist-test-slave-gl2g"
server_key: "bd56266a382e2e7006126ec68bb4e18a"
server_key_iv: "d9b5a3bf746a0e04800e2b8f6b933499"
server_key_version: "encryptionkey@0"
I20260620 22:02:43.654564 27519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "72c3c86f4fc24c89a40c042dd568e9b0"
format_stamp: "Formatted at 2026-06-20 22:02:43 on dist-test-slave-gl2g"
server_key: "bd56266a382e2e7006126ec68bb4e18a"
server_key_iv: "d9b5a3bf746a0e04800e2b8f6b933499"
server_key_version: "encryptionkey@0"
I20260620 22:02:43.655896 27519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:02:43.656610 27534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:43.656780 27519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:02:43.656881 27519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "72c3c86f4fc24c89a40c042dd568e9b0"
format_stamp: "Formatted at 2026-06-20 22:02:43 on dist-test-slave-gl2g"
server_key: "bd56266a382e2e7006126ec68bb4e18a"
server_key_iv: "d9b5a3bf746a0e04800e2b8f6b933499"
server_key_version: "encryptionkey@0"
I20260620 22:02:43.656963 27519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:43.660357 27519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:43.662175 27519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:43.662339 27519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:43.666328 27519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33765
I20260620 22:02:43.666353 27586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33765 every 8 connection(s)
I20260620 22:02:43.666760 27519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:43.667802 27587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:43.670092 27587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0: Bootstrap starting.
I20260620 22:02:43.670742 27587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:43.671039 27587 log.cc:824] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:43.671721 27587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0: No bootstrap required, opened a new log
I20260620 22:02:43.672425 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 27519
I20260620 22:02:43.672549 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:43.672729 7110 external_mini_cluster.cc:1428] Setting key 977c0c401204045a2c3844eca19ecba0
I20260620 22:02:43.673774 27587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c3c86f4fc24c89a40c042dd568e9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } }
I20260620 22:02:43.673965 27587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:43.674010 27587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c3c86f4fc24c89a40c042dd568e9b0, State: Initialized, Role: FOLLOWER
I20260620 22:02:43.674530 27587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c3c86f4fc24c89a40c042dd568e9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } }
I20260620 22:02:43.674675 27587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:43.674715 27587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:43.674767 27587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:43.675278 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:43.673255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60869 (local address 127.0.0.1:33765)
0620 22:02:43.673469 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:43.673478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:43.673504 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:02:43.673751 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:43.673758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:43.673784 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:43.673900 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:43.674589 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:43.674592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:43.674592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:43.674628 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:43.674692 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:43.674711 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:43.674786 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:43.674974 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 22:02:43.675025 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
I20260620 22:02:43.676009 27587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c3c86f4fc24c89a40c042dd568e9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } }
I20260620 22:02:43.676115 27587 leader_election.cc:302] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72c3c86f4fc24c89a40c042dd568e9b0; no voters:
I20260620 22:02:43.676265 27587 leader_election.cc:288] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:43.676251 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:43.676393 27592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:43.676492 27587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:43.676693 27592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [term 1 LEADER]: Becoming Leader. State: Replica: 72c3c86f4fc24c89a40c042dd568e9b0, State: Running, Role: LEADER
I20260620 22:02:43.676940 27592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c3c86f4fc24c89a40c042dd568e9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } }
I20260620 22:02:43.677626 27594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72c3c86f4fc24c89a40c042dd568e9b0. Latest consensus state: current_term: 1 leader_uuid: "72c3c86f4fc24c89a40c042dd568e9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c3c86f4fc24c89a40c042dd568e9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } } }
I20260620 22:02:43.677800 27593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72c3c86f4fc24c89a40c042dd568e9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c3c86f4fc24c89a40c042dd568e9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } } }
I20260620 22:02:43.677899 27593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:43.677745 27594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:43.678303 27611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:43.683400 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:43.677879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55387 (local address 127.0.0.1:33765)
0620 22:02:43.677910 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:43.677913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:43.678140 (+ 227us) server_negotiation.cc:407] Connection header received
0620 22:02:43.678233 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:43.678234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:43.678240 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:43.678333 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:43.683154 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:43.683156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:43.683156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:43.683161 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:43.683179 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:43.683185 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:43.683217 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:43.683316 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 22:02:43.683331 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 22:02:43.686235 27611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:43.688099 27611 catalog_manager.cc:1347] Generated new cluster ID: 5dd696497d1643818b2c47ff67e8a499
I20260620 22:02:43.688161 27611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:43.709451 27611 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:43.709681 27611 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:43.715734 27611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72c3c86f4fc24c89a40c042dd568e9b0: Generated new TSK 0
I20260620 22:02:43.715956 27611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:43.717912 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 27519
2026-06-20T22:02:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:43 dist-test-slave-gl2g krb5kdc[27618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:43 dist-test-slave-gl2g krb5kdc[27618](info): set up 2 sockets
Jun 20 22:02:43 dist-test-slave-gl2g krb5kdc[27618](info): commencing operation
krb5kdc: starting...
W20260620 22:02:44.751238 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:44 dist-test-slave-gl2g krb5kdc[27618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:44.869299 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:44.951768 27631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:44.952001 27631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:44.952025 27631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:44.953212 27631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:44.953258 27631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:44.953274 27631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:44.953285 27631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:44.953339 27631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:44.954918 27631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:44.955163 27631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:44.955401 27631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:02:44.960402 27631 server_base.cc:1034] running on GCE node
W20260620 22:02:44.960364 27637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:44.960369 27639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:44.960421 27636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:44.960807 27631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:44.961130 27631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:44.962301 27631 hybrid_clock.cc:648] HybridClock initialized: now 1781992964962287 us; error 28 us; skew 500 ppm
Jun 20 22:02:44 dist-test-slave-gl2g krb5kdc[27618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:44.964941 27631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:44.965435 27631 webserver.cc:460] Webserver started at http://127.0.0.1:36657/ using document root <none> and password file <none>
I20260620 22:02:44.965682 27631 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:44.965727 27631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:44.965893 27631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:44.967039 27631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "eaaa3a9731334859af164a4f6481b772"
format_stamp: "Formatted at 2026-06-20 22:02:44 on dist-test-slave-gl2g"
server_key: "0874a5efe4aa6bcca2a99a79e8573488"
server_key_iv: "a6c9e760a6eed6f24131f1390b095cf1"
server_key_version: "encryptionkey@0"
I20260620 22:02:44.967396 27631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "eaaa3a9731334859af164a4f6481b772"
format_stamp: "Formatted at 2026-06-20 22:02:44 on dist-test-slave-gl2g"
server_key: "0874a5efe4aa6bcca2a99a79e8573488"
server_key_iv: "a6c9e760a6eed6f24131f1390b095cf1"
server_key_version: "encryptionkey@0"
I20260620 22:02:44.968732 27631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:44.969521 27646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:44.969822 27631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:44.969926 27631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "eaaa3a9731334859af164a4f6481b772"
format_stamp: "Formatted at 2026-06-20 22:02:44 on dist-test-slave-gl2g"
server_key: "0874a5efe4aa6bcca2a99a79e8573488"
server_key_iv: "a6c9e760a6eed6f24131f1390b095cf1"
server_key_version: "encryptionkey@0"
I20260620 22:02:44.970003 27631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:44.980629 27631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:44.983078 27631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:44.983259 27631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:44.987627 27631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37211
I20260620 22:02:44.987671 27698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37211 every 8 connection(s)
I20260620 22:02:44.988063 27631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:44.989090 27699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:44.991466 27699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772: Bootstrap starting.
I20260620 22:02:44.992014 27699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:44.992251 27699 log.cc:824] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:44.992928 27699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772: No bootstrap required, opened a new log
I20260620 22:02:44.994508 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 27631
I20260620 22:02:44.994658 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:44.994681 27699 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaa3a9731334859af164a4f6481b772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } }
I20260620 22:02:44.994871 7110 external_mini_cluster.cc:1428] Setting key 225e8fc5ce8041e68883b053c27d1ea2
I20260620 22:02:44.994870 27699 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:44.994910 27699 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaa3a9731334859af164a4f6481b772, State: Initialized, Role: FOLLOWER
I20260620 22:02:44.995067 27699 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaa3a9731334859af164a4f6481b772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } }
I20260620 22:02:44.995139 27699 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:44.995188 27699 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:44.995244 27699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:44.995957 27699 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaa3a9731334859af164a4f6481b772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } }
I20260620 22:02:44.996081 27699 leader_election.cc:302] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaaa3a9731334859af164a4f6481b772; no voters:
I20260620 22:02:44.996272 27699 leader_election.cc:288] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:44.996304 27704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:44.996444 27704 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [term 1 LEADER]: Becoming Leader. State: Replica: eaaa3a9731334859af164a4f6481b772, State: Running, Role: LEADER
I20260620 22:02:44.996537 27704 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaa3a9731334859af164a4f6481b772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } }
I20260620 22:02:44.996562 27699 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:44.996905 27706 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaaa3a9731334859af164a4f6481b772. Latest consensus state: current_term: 1 leader_uuid: "eaaa3a9731334859af164a4f6481b772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaa3a9731334859af164a4f6481b772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } } }
I20260620 22:02:44.997082 27706 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:44.996881 27705 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaaa3a9731334859af164a4f6481b772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaaa3a9731334859af164a4f6481b772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } } }
I20260620 22:02:44.997679 27705 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:44.997689 27708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:44.998236 27708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:44.999897 27708 catalog_manager.cc:1347] Generated new cluster ID: 387cf4a2c3794df990532db89945d95c
I20260620 22:02:44.999950 27708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:02:44 dist-test-slave-gl2g krb5kdc[27618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:02:45.007198 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:44.995372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34215 (local address 127.0.0.1:37211)
0620 22:02:44.995542 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:44.995552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:44.995602 (+ 50us) server_negotiation.cc:407] Connection header received
0620 22:02:44.995935 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:44.995941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:44.995989 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:44.996147 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:44.997057 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:44.998466 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:44.999472 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:45.000079 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:45.000192 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:45.003120 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:45.003124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:45.003125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:45.003138 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:45.005368 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:45.005761 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:45.005765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:45.005765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:45.005923 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:45.006136 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:45.006136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:45.006137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:45.006491 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:45.006690 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:45.006784 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 22:02:45.006967 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260620 22:02:45.008237 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:45.016911 27708 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:45.017316 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:45.009760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46641 (local address 127.0.0.1:37211)
0620 22:02:45.009797 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:45.009799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:45.009872 (+ 73us) server_negotiation.cc:407] Connection header received
0620 22:02:45.009928 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:45.009928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:45.009938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:45.010004 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:45.010625 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:45.011393 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:45.012352 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:45.012686 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:45.012759 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:45.014111 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:45.014112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:45.014112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:45.014119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:45.015740 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:45.016127 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:45.016128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:45.016129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:45.016188 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:45.016451 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:45.016452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:45.016452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:45.016596 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:45.016696 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:45.017136 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 22:02:45.017212 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:02:45.017643 27708 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:45.024721 27708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaaa3a9731334859af164a4f6481b772: Generated new TSK 0
I20260620 22:02:45.024940 27708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:45.031463 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 27631
2026-06-20T22:02:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:45 dist-test-slave-gl2g krb5kdc[27730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:45 dist-test-slave-gl2g krb5kdc[27730](info): set up 2 sockets
Jun 20 22:02:45 dist-test-slave-gl2g krb5kdc[27730](info): commencing operation
krb5kdc: starting...
W20260620 22:02:46.066814 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:46 dist-test-slave-gl2g krb5kdc[27730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:46.191045 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:46.273974 27743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:46.274215 27743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:46.274241 27743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:46.275565 27743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:46.275620 27743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:46.275635 27743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:46.275650 27743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:46.275662 27743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:46.277282 27743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:46.277558 27743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:46.277823 27743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:46.282912 27749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:46.282907 27751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:46.282908 27748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:46.283151 27743 server_base.cc:1034] running on GCE node
I20260620 22:02:46.283535 27743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:46.283869 27743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:46.285027 27743 hybrid_clock.cc:648] HybridClock initialized: now 1781992966285006 us; error 41 us; skew 500 ppm
Jun 20 22:02:46 dist-test-slave-gl2g krb5kdc[27730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:46.287770 27743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:46.288286 27743 webserver.cc:460] Webserver started at http://127.0.0.1:32827/ using document root <none> and password file <none>
I20260620 22:02:46.288542 27743 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:46.288590 27743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:46.288765 27743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:46.289862 27743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "21073831c2bf49169af947d808370430"
format_stamp: "Formatted at 2026-06-20 22:02:46 on dist-test-slave-gl2g"
server_key: "15d3a53fb5efb053a0924524bf26b0a3"
server_key_iv: "232230cabef92f76ef85a7dbd210952e"
server_key_version: "encryptionkey@0"
I20260620 22:02:46.290251 27743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "21073831c2bf49169af947d808370430"
format_stamp: "Formatted at 2026-06-20 22:02:46 on dist-test-slave-gl2g"
server_key: "15d3a53fb5efb053a0924524bf26b0a3"
server_key_iv: "232230cabef92f76ef85a7dbd210952e"
server_key_version: "encryptionkey@0"
I20260620 22:02:46.291735 27743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:02:46.292527 27758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:46.292714 27743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:46.292824 27743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "21073831c2bf49169af947d808370430"
format_stamp: "Formatted at 2026-06-20 22:02:46 on dist-test-slave-gl2g"
server_key: "15d3a53fb5efb053a0924524bf26b0a3"
server_key_iv: "232230cabef92f76ef85a7dbd210952e"
server_key_version: "encryptionkey@0"
I20260620 22:02:46.292903 27743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:46.305265 27743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:46.307782 27743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:46.307960 27743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:46.312288 27743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33049
I20260620 22:02:46.312335 27810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33049 every 8 connection(s)
I20260620 22:02:46.312721 27743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:46.313716 27811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:46.316084 27811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430: Bootstrap starting.
I20260620 22:02:46.316494 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 27743
I20260620 22:02:46.316594 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:46.316790 27811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:46.316807 7110 external_mini_cluster.cc:1428] Setting key 3ff98f159fc59a798ab86f0e950c9a89
I20260620 22:02:46.317088 27811 log.cc:824] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:46.317893 27811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430: No bootstrap required, opened a new log
I20260620 22:02:46.320417 27811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21073831c2bf49169af947d808370430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } }
I20260620 22:02:46.320573 27811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:46.320597 27811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21073831c2bf49169af947d808370430, State: Initialized, Role: FOLLOWER
I20260620 22:02:46.320729 27811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21073831c2bf49169af947d808370430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } }
I20260620 22:02:46.320807 27811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:46.320856 27811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:46.320907 27811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:46.321628 27811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21073831c2bf49169af947d808370430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } }
I20260620 22:02:46.321738 27811 leader_election.cc:302] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21073831c2bf49169af947d808370430; no voters:
I20260620 22:02:46.321908 27811 leader_election.cc:288] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:46.321969 27816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:46.322153 27816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [term 1 LEADER]: Becoming Leader. State: Replica: 21073831c2bf49169af947d808370430, State: Running, Role: LEADER
I20260620 22:02:46.322173 27811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:46.322250 27816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21073831c2bf49169af947d808370430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } }
I20260620 22:02:46.322613 27816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21073831c2bf49169af947d808370430. Latest consensus state: current_term: 1 leader_uuid: "21073831c2bf49169af947d808370430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21073831c2bf49169af947d808370430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } } }
I20260620 22:02:46.322916 27816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:46.322614 27817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21073831c2bf49169af947d808370430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21073831c2bf49169af947d808370430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } } }
I20260620 22:02:46.323371 27817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:46.323577 27821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:46.323843 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:46.317324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56113 (local address 127.0.0.1:33049)
0620 22:02:46.317498 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:46.317509 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:46.317559 (+ 50us) server_negotiation.cc:407] Connection header received
0620 22:02:46.318168 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:46.318177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:46.318240 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:46.318397 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:46.318668 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:46.320175 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:46.321265 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:46.321917 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:46.322399 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:46.322903 (+ 504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:46.322908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:46.322908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:46.323136 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:46.323178 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:46.323194 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:46.323261 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:46.323495 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 22:02:46.323569 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260620 22:02:46.324131 27821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:46.324759 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:46.326917 27821 catalog_manager.cc:1347] Generated new cluster ID: 3d82609afa1e408bab04f921711a420e
I20260620 22:02:46.326975 27821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:46.329519 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:46.326089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34037 (local address 127.0.0.1:33049)
0620 22:02:46.326122 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:46.326124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:46.326776 (+ 652us) server_negotiation.cc:407] Connection header received
0620 22:02:46.326870 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:46.326871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:46.326883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:46.326913 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:46.327120 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:46.327592 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:46.328309 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:46.328523 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:46.328599 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:46.329293 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:46.329295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:46.329295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:46.329300 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:46.329315 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:46.329319 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:46.329361 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:46.329416 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 22:02:46.329439 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:02:46.341794 27821 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:46.342558 27821 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:46.353266 27821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21073831c2bf49169af947d808370430: Generated new TSK 0
I20260620 22:02:46.353504 27821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:46.356809 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 27743
2026-06-20T22:02:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:46 dist-test-slave-gl2g krb5kdc[27842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:46 dist-test-slave-gl2g krb5kdc[27842](info): set up 2 sockets
Jun 20 22:02:46 dist-test-slave-gl2g krb5kdc[27842](info): commencing operation
krb5kdc: starting...
W20260620 22:02:47.386699 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:47 dist-test-slave-gl2g krb5kdc[27842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:47.503149 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:47.585853 27855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:47.586091 27855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:47.586117 27855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:47.587267 27855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:47.587308 27855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:47.587329 27855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:47.587381 27855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:47.587400 27855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:47.588869 27855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:47.589114 27855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:47.589342 27855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:47.594110 27861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:47.594166 27860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:47.594255 27863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:47.594223 27855 server_base.cc:1034] running on GCE node
I20260620 22:02:47.594635 27855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:47.594923 27855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:47.596081 27855 hybrid_clock.cc:648] HybridClock initialized: now 1781992967596066 us; error 47 us; skew 500 ppm
Jun 20 22:02:47 dist-test-slave-gl2g krb5kdc[27842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:47.602977 27855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:47.603363 27855 webserver.cc:460] Webserver started at http://127.0.0.1:44687/ using document root <none> and password file <none>
I20260620 22:02:47.603538 27855 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:47.603628 27855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:47.603785 27855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:47.604801 27855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "ca2e644ca80b4334842b33d0d4d44ad2"
format_stamp: "Formatted at 2026-06-20 22:02:47 on dist-test-slave-gl2g"
server_key: "90fea2f88bceb4b4620cd5a7399bde05"
server_key_iv: "6d31554fe1ca1227292c25b241bea250"
server_key_version: "encryptionkey@0"
I20260620 22:02:47.605094 27855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "ca2e644ca80b4334842b33d0d4d44ad2"
format_stamp: "Formatted at 2026-06-20 22:02:47 on dist-test-slave-gl2g"
server_key: "90fea2f88bceb4b4620cd5a7399bde05"
server_key_iv: "6d31554fe1ca1227292c25b241bea250"
server_key_version: "encryptionkey@0"
I20260620 22:02:47.606387 27855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:47.607194 27870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:47.607434 27855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:02:47.607549 27855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "ca2e644ca80b4334842b33d0d4d44ad2"
format_stamp: "Formatted at 2026-06-20 22:02:47 on dist-test-slave-gl2g"
server_key: "90fea2f88bceb4b4620cd5a7399bde05"
server_key_iv: "6d31554fe1ca1227292c25b241bea250"
server_key_version: "encryptionkey@0"
I20260620 22:02:47.607628 27855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:47.611011 27855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:47.613271 27855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:47.613432 27855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:47.617236 27855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34977
I20260620 22:02:47.617297 27922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34977 every 8 connection(s)
I20260620 22:02:47.617636 27855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:47.618073 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 27855
I20260620 22:02:47.618211 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:47.618417 7110 external_mini_cluster.cc:1428] Setting key bad488d2a1e49e9e4826ff8d13b1f42f
I20260620 22:02:47.618759 27923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:47.620985 27926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:47.618961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50419 (local address 127.0.0.1:34977)
0620 22:02:47.619132 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:47.619141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:47.619174 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:02:47.619783 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:47.619790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:47.619814 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:47.619926 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:47.620526 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:47.620530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:47.620530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:47.620533 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:47.620569 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:47.620579 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:47.620633 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:47.620736 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 22:02:47.620789 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260620 22:02:47.621966 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:47.622076 27923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2: Bootstrap starting.
I20260620 22:02:47.622829 27923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:47.623176 27923 log.cc:824] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:47.624142 27923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2: No bootstrap required, opened a new log
I20260620 22:02:47.624548 27926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:47.623574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36955 (local address 127.0.0.1:34977)
0620 22:02:47.623609 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:47.623611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:47.623689 (+ 78us) server_negotiation.cc:407] Connection header received
0620 22:02:47.623759 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:47.623760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:47.623763 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:47.623811 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:47.624358 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:47.624359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:47.624359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:47.624363 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:47.624377 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:47.624381 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:47.624448 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:47.624481 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 22:02:47.624494 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:02:47.626209 27923 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2e644ca80b4334842b33d0d4d44ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260620 22:02:47.626348 27923 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:47.626371 27923 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2e644ca80b4334842b33d0d4d44ad2, State: Initialized, Role: FOLLOWER
I20260620 22:02:47.626492 27923 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2e644ca80b4334842b33d0d4d44ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260620 22:02:47.626580 27923 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:47.626657 27923 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:47.626709 27923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:47.627240 27923 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2e644ca80b4334842b33d0d4d44ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260620 22:02:47.627359 27923 leader_election.cc:302] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2e644ca80b4334842b33d0d4d44ad2; no voters:
I20260620 22:02:47.627583 27923 leader_election.cc:288] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:47.627640 27933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:47.627818 27933 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [term 1 LEADER]: Becoming Leader. State: Replica: ca2e644ca80b4334842b33d0d4d44ad2, State: Running, Role: LEADER
I20260620 22:02:47.627830 27923 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:47.627995 27933 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2e644ca80b4334842b33d0d4d44ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260620 22:02:47.628383 27935 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2e644ca80b4334842b33d0d4d44ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2e644ca80b4334842b33d0d4d44ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } } }
I20260620 22:02:47.628388 27934 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2e644ca80b4334842b33d0d4d44ad2. Latest consensus state: current_term: 1 leader_uuid: "ca2e644ca80b4334842b33d0d4d44ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2e644ca80b4334842b33d0d4d44ad2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } } }
I20260620 22:02:47.628530 27935 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:47.628547 27934 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:47.628770 27942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:47.629252 27942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:47.630918 27942 catalog_manager.cc:1347] Generated new cluster ID: 93d367c076da49fa8b391b362aa78b69
I20260620 22:02:47.630959 27942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:47.645617 27942 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:47.645802 27942 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:47.647639 27942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca2e644ca80b4334842b33d0d4d44ad2: Generated new TSK 0
I20260620 22:02:47.647819 27942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:47.651648 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 27855
2026-06-20T22:02:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:47 dist-test-slave-gl2g krb5kdc[27954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:47 dist-test-slave-gl2g krb5kdc[27954](info): set up 2 sockets
Jun 20 22:02:47 dist-test-slave-gl2g krb5kdc[27954](info): commencing operation
krb5kdc: starting...
W20260620 22:02:48.690830 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:48 dist-test-slave-gl2g krb5kdc[27954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:48.809881 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:48.892122 27967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:48.892350 27967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:48.892375 27967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:48.893581 27967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:48.893631 27967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:48.893646 27967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:48.893661 27967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:48.893672 27967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:48.895293 27967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:48.895567 27967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:48.895807 27967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:48.900976 27973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:48.901041 27967 server_base.cc:1034] running on GCE node
W20260620 22:02:48.900980 27972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:48.900976 27975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:48.901413 27967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:48.901724 27967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:48.902865 27967 hybrid_clock.cc:648] HybridClock initialized: now 1781992968902844 us; error 36 us; skew 500 ppm
Jun 20 22:02:48 dist-test-slave-gl2g krb5kdc[27954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:48.905561 27967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:48.906059 27967 webserver.cc:460] Webserver started at http://127.0.0.1:33379/ using document root <none> and password file <none>
I20260620 22:02:48.906292 27967 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:48.906340 27967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:48.906479 27967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:48.907640 27967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "ea9676926c7543559ba481eee2cd9185"
format_stamp: "Formatted at 2026-06-20 22:02:48 on dist-test-slave-gl2g"
server_key: "874b37a0b9d15e8ab6bcdf48ffe54127"
server_key_iv: "1a310fad3101cd432f4a13752435b7dc"
server_key_version: "encryptionkey@0"
I20260620 22:02:48.908010 27967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "ea9676926c7543559ba481eee2cd9185"
format_stamp: "Formatted at 2026-06-20 22:02:48 on dist-test-slave-gl2g"
server_key: "874b37a0b9d15e8ab6bcdf48ffe54127"
server_key_iv: "1a310fad3101cd432f4a13752435b7dc"
server_key_version: "encryptionkey@0"
I20260620 22:02:48.909390 27967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:02:48.910271 27982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:48.910483 27967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:48.910610 27967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "ea9676926c7543559ba481eee2cd9185"
format_stamp: "Formatted at 2026-06-20 22:02:48 on dist-test-slave-gl2g"
server_key: "874b37a0b9d15e8ab6bcdf48ffe54127"
server_key_iv: "1a310fad3101cd432f4a13752435b7dc"
server_key_version: "encryptionkey@0"
I20260620 22:02:48.910727 27967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:48.918253 27967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:48.920275 27967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:48.920435 27967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:48.924611 27967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33135
I20260620 22:02:48.924662 28034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33135 every 8 connection(s)
I20260620 22:02:48.925024 27967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:48.925985 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:48.928279 28035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185: Bootstrap starting.
I20260620 22:02:48.928803 28035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:48.929030 28035 log.cc:824] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:48.929680 28035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185: No bootstrap required, opened a new log
I20260620 22:02:48.931303 28035 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9676926c7543559ba481eee2cd9185" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } }
I20260620 22:02:48.931448 28035 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:48.931473 28035 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9676926c7543559ba481eee2cd9185, State: Initialized, Role: FOLLOWER
I20260620 22:02:48.931610 28035 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9676926c7543559ba481eee2cd9185" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } }
I20260620 22:02:48.931692 28035 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:48.931715 28035 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:48.931744 28035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:48.932214 28035 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9676926c7543559ba481eee2cd9185" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } }
I20260620 22:02:48.932296 28035 leader_election.cc:302] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9676926c7543559ba481eee2cd9185; no voters:
I20260620 22:02:48.932466 28035 leader_election.cc:288] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:48.932497 28038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:48.932614 28038 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [term 1 LEADER]: Becoming Leader. State: Replica: ea9676926c7543559ba481eee2cd9185, State: Running, Role: LEADER
I20260620 22:02:48.932729 28035 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:48.932752 28038 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9676926c7543559ba481eee2cd9185" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } }
I20260620 22:02:48.933211 28039 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9676926c7543559ba481eee2cd9185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9676926c7543559ba481eee2cd9185" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } } }
I20260620 22:02:48.933329 28040 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9676926c7543559ba481eee2cd9185. Latest consensus state: current_term: 1 leader_uuid: "ea9676926c7543559ba481eee2cd9185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9676926c7543559ba481eee2cd9185" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33135 } } }
I20260620 22:02:48.933480 28040 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:48.933380 28039 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:48.933861 28047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:48.934406 28047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:48.934725 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 27967
I20260620 22:02:48.934844 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:48.935014 7110 external_mini_cluster.cc:1428] Setting key ad611d8a93fb74a09c96f562d5cf6b0d
I20260620 22:02:48.936710 28047 catalog_manager.cc:1347] Generated new cluster ID: 1afb2388233747648c3827bcf3afa8d1
I20260620 22:02:48.936764 28047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:02:48 dist-test-slave-gl2g krb5kdc[27954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:02:48.947192 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:48.935641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:37887 (local address 127.0.0.1:33135)
0620 22:02:48.936091 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:48.936098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:48.936120 (+ 22us) server_negotiation.cc:407] Connection header received
0620 22:02:48.936208 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:48.936214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:48.936257 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:48.936400 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:48.937155 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:48.938709 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:48.939745 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:48.940403 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:48.940514 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:48.943344 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:48.943348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:48.943349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:48.943362 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:48.945415 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:48.945847 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:48.945851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:48.945851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:48.945971 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:48.946198 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:48.946199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:48.946199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:48.946518 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:48.946705 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:48.946819 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 22:02:48.946927 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":69,"threads_started":1}
I20260620 22:02:48.948215 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:48.952762 28047 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:48.953258 28047 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:48.960068 28047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9676926c7543559ba481eee2cd9185: Generated new TSK 0
I20260620 22:02:48.960285 28047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:48.962391 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:48.949770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:60073 (local address 127.0.0.1:33135)
0620 22:02:48.949810 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:48.949812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:48.952468 (+ 2656us) server_negotiation.cc:407] Connection header received
0620 22:02:48.952505 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:48.952506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:48.952519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:48.952570 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:48.955267 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:48.956395 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:48.957411 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:48.957744 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:48.957819 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:48.959359 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:48.959361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:48.959361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:48.959369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:48.961171 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:48.961560 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:48.961564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:48.961564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:48.961620 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:48.961861 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:48.961862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:48.961862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:48.961978 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:48.962090 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:48.962212 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 22:02:48.962277 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:02:48.964227 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 27967
W20260620 22:02:48.968739 27952 connection.cc:535] client connection to 127.0.0.1:33135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33135): Connection reset by peer (error 104)
2026-06-20T22:02:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:49 dist-test-slave-gl2g krb5kdc[28066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:49 dist-test-slave-gl2g krb5kdc[28066](info): set up 2 sockets
Jun 20 22:02:49 dist-test-slave-gl2g krb5kdc[28066](info): commencing operation
krb5kdc: starting...
W20260620 22:02:49.994735 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:50 dist-test-slave-gl2g krb5kdc[28066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:50.109405 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:50.192612 28079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:50.192814 28079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:50.192836 28079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:50.194010 28079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:50.194053 28079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:50.194068 28079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:50.194080 28079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:50.194144 28079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:50.195613 28079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:50.195875 28079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:50.196110 28079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:50.200806 28084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:50.200938 28079 server_base.cc:1034] running on GCE node
W20260620 22:02:50.200848 28087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:50.200836 28085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:50.201282 28079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:50.201576 28079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:50.202737 28079 hybrid_clock.cc:648] HybridClock initialized: now 1781992970202720 us; error 50 us; skew 500 ppm
Jun 20 22:02:50 dist-test-slave-gl2g krb5kdc[28066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:50.205173 28079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:50.205632 28079 webserver.cc:460] Webserver started at http://127.0.0.1:38919/ using document root <none> and password file <none>
I20260620 22:02:50.205864 28079 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:50.205940 28079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:50.206089 28079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:50.207191 28079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "100b9662604643b5a2ee7f1834a84d78"
format_stamp: "Formatted at 2026-06-20 22:02:50 on dist-test-slave-gl2g"
server_key: "bb6173a43ad6206cf994717b9582009f"
server_key_iv: "19474591aa00c107b167af39ff9bc58a"
server_key_version: "encryptionkey@0"
I20260620 22:02:50.207540 28079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "100b9662604643b5a2ee7f1834a84d78"
format_stamp: "Formatted at 2026-06-20 22:02:50 on dist-test-slave-gl2g"
server_key: "bb6173a43ad6206cf994717b9582009f"
server_key_iv: "19474591aa00c107b167af39ff9bc58a"
server_key_version: "encryptionkey@0"
I20260620 22:02:50.208850 28079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:02:50.209534 28094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:50.209707 28079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:50.209812 28079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "100b9662604643b5a2ee7f1834a84d78"
format_stamp: "Formatted at 2026-06-20 22:02:50 on dist-test-slave-gl2g"
server_key: "bb6173a43ad6206cf994717b9582009f"
server_key_iv: "19474591aa00c107b167af39ff9bc58a"
server_key_version: "encryptionkey@0"
I20260620 22:02:50.209885 28079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:50.220369 28079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:50.222649 28079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:50.222807 28079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:50.226830 28079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40839
I20260620 22:02:50.226871 28146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40839 every 8 connection(s)
I20260620 22:02:50.227193 28079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:50.228076 28147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:50.230253 28147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78: Bootstrap starting.
I20260620 22:02:50.231014 28147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:50.231304 28147 log.cc:824] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:50.231971 28147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78: No bootstrap required, opened a new log
I20260620 22:02:50.233512 28147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b9662604643b5a2ee7f1834a84d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260620 22:02:50.233714 28147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:50.233767 28147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100b9662604643b5a2ee7f1834a84d78, State: Initialized, Role: FOLLOWER
I20260620 22:02:50.233938 28147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b9662604643b5a2ee7f1834a84d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260620 22:02:50.233990 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 28079
I20260620 22:02:50.234042 28147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:50.234086 28147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:50.234092 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:50.234140 28147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:50.234306 7110 external_mini_cluster.cc:1428] Setting key 914b598e10fc0a46d3be5b51bfa82ab5
I20260620 22:02:50.234830 28147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b9662604643b5a2ee7f1834a84d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260620 22:02:50.234975 28147 leader_election.cc:302] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 100b9662604643b5a2ee7f1834a84d78; no voters:
I20260620 22:02:50.235152 28147 leader_election.cc:288] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:50.235181 28152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:50.235286 28152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [term 1 LEADER]: Becoming Leader. State: Replica: 100b9662604643b5a2ee7f1834a84d78, State: Running, Role: LEADER
I20260620 22:02:50.235407 28147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:50.235422 28152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b9662604643b5a2ee7f1834a84d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260620 22:02:50.236802 28153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 100b9662604643b5a2ee7f1834a84d78. Latest consensus state: current_term: 1 leader_uuid: "100b9662604643b5a2ee7f1834a84d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b9662604643b5a2ee7f1834a84d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } } }
I20260620 22:02:50.236951 28154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "100b9662604643b5a2ee7f1834a84d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b9662604643b5a2ee7f1834a84d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } } }
I20260620 22:02:50.237017 28153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:50.237025 28154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78 [sys.catalog]: This master's current role is: LEADER
W20260620 22:02:50.237319 28167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:02:50.237371 28167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:02:50.237414 28169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:50.237556 28169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:50.239125 28169 catalog_manager.cc:1347] Generated new cluster ID: cb15b202318145baa94c7b3c1f140bc5
I20260620 22:02:50.239178 28169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:50.243609 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:50.234764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:34923 (local address 127.0.0.1:40839)
0620 22:02:50.234941 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:50.234951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:50.234986 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:02:50.235102 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:50.235109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:50.235163 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:50.235310 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:50.235643 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:50.237110 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:50.238026 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:50.238659 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:50.238774 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:50.242876 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:50.242880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:50.242880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:50.243129 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:50.243169 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:50.243184 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:50.243286 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:50.243377 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:02:50.243405 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260620 22:02:50.244532 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:50.249508 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:50.246052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:40681 (local address 127.0.0.1:40839)
0620 22:02:50.246083 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:50.246085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:50.246216 (+ 131us) server_negotiation.cc:407] Connection header received
0620 22:02:50.246279 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:50.246279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:50.246290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:50.246341 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:50.246474 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:50.247217 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:50.248250 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:50.248541 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:50.248601 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:50.249262 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:50.249263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:50.249264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:50.249267 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:50.249280 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:50.249284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:50.249336 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:50.249426 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 22:02:50.249438 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":17}
I20260620 22:02:50.262068 28169 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:50.262856 28169 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:50.276500 28169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 100b9662604643b5a2ee7f1834a84d78: Generated new TSK 0
I20260620 22:02:50.276711 28169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:50.286302 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 28079
W20260620 22:02:50.290736 28064 connection.cc:535] client connection to 127.0.0.1:40839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40839): Connection reset by peer (error 104)
2026-06-20T22:02:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:50 dist-test-slave-gl2g krb5kdc[28178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:50 dist-test-slave-gl2g krb5kdc[28178](info): set up 2 sockets
Jun 20 22:02:50 dist-test-slave-gl2g krb5kdc[28178](info): commencing operation
krb5kdc: starting...
W20260620 22:02:51.322777 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:51 dist-test-slave-gl2g krb5kdc[28178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:51.435316 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:51.514935 28191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:51.515141 28191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:51.515164 28191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:51.516305 28191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:51.516345 28191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:51.516360 28191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:51.516372 28191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:51.516386 28191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:51.517874 28191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:51.518122 28191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:51.518332 28191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:51.523098 28196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:51.523070 28199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:51.523109 28191 server_base.cc:1034] running on GCE node
W20260620 22:02:51.523078 28197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:51.523561 28191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:51.523836 28191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:51.524999 28191 hybrid_clock.cc:648] HybridClock initialized: now 1781992971524987 us; error 30 us; skew 500 ppm
Jun 20 22:02:51 dist-test-slave-gl2g krb5kdc[28178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:51.527586 28191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:51.528007 28191 webserver.cc:460] Webserver started at http://127.0.0.1:41061/ using document root <none> and password file <none>
I20260620 22:02:51.528241 28191 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:51.528309 28191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:51.528467 28191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:51.529507 28191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "f10599e6b05a47138f811c6f4a5c87e3"
format_stamp: "Formatted at 2026-06-20 22:02:51 on dist-test-slave-gl2g"
server_key: "342f17b279b528e371503fa2a9c5491f"
server_key_iv: "eb71981610a92b5540cd1434a14250ed"
server_key_version: "encryptionkey@0"
I20260620 22:02:51.529870 28191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "f10599e6b05a47138f811c6f4a5c87e3"
format_stamp: "Formatted at 2026-06-20 22:02:51 on dist-test-slave-gl2g"
server_key: "342f17b279b528e371503fa2a9c5491f"
server_key_iv: "eb71981610a92b5540cd1434a14250ed"
server_key_version: "encryptionkey@0"
I20260620 22:02:51.531224 28191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:02:51.531939 28206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:51.532123 28191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:51.532209 28191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f10599e6b05a47138f811c6f4a5c87e3"
format_stamp: "Formatted at 2026-06-20 22:02:51 on dist-test-slave-gl2g"
server_key: "342f17b279b528e371503fa2a9c5491f"
server_key_iv: "eb71981610a92b5540cd1434a14250ed"
server_key_version: "encryptionkey@0"
I20260620 22:02:51.532286 28191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:51.535527 28191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:51.537750 28191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:51.537926 28191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:51.542042 28191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42975
I20260620 22:02:51.542074 28258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42975 every 8 connection(s)
I20260620 22:02:51.542407 28191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:51.543390 28259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:51.545552 28259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3: Bootstrap starting.
I20260620 22:02:51.546152 28259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:51.546432 28259 log.cc:824] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:51.547191 28259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3: No bootstrap required, opened a new log
I20260620 22:02:51.548728 28259 raft_consensus.cc:348] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10599e6b05a47138f811c6f4a5c87e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42975 } }
I20260620 22:02:51.548861 28259 raft_consensus.cc:374] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:51.548882 28259 raft_consensus.cc:729] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10599e6b05a47138f811c6f4a5c87e3, State: Initialized, Role: FOLLOWER
I20260620 22:02:51.549033 28259 consensus_queue.cc:260] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10599e6b05a47138f811c6f4a5c87e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42975 } }
I20260620 22:02:51.549124 28259 raft_consensus.cc:388] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:51.549147 28259 raft_consensus.cc:482] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:51.549212 28259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:51.549546 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 28191
I20260620 22:02:51.549664 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:51.549860 7110 external_mini_cluster.cc:1428] Setting key 1e053d98539f02c95b7a158883ef6335
I20260620 22:02:51.549943 28259 raft_consensus.cc:504] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10599e6b05a47138f811c6f4a5c87e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42975 } }
I20260620 22:02:51.550055 28259 leader_election.cc:302] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f10599e6b05a47138f811c6f4a5c87e3; no voters:
I20260620 22:02:51.550239 28259 leader_election.cc:288] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:51.550297 28262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:51.550537 28259 sys_catalog.cc:564] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:51.550551 28262 raft_consensus.cc:686] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [term 1 LEADER]: Becoming Leader. State: Replica: f10599e6b05a47138f811c6f4a5c87e3, State: Running, Role: LEADER
I20260620 22:02:51.550741 28262 consensus_queue.cc:237] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10599e6b05a47138f811c6f4a5c87e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42975 } }
I20260620 22:02:51.551378 28266 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f10599e6b05a47138f811c6f4a5c87e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10599e6b05a47138f811c6f4a5c87e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42975 } } }
I20260620 22:02:51.551517 28266 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:51.551687 28264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f10599e6b05a47138f811c6f4a5c87e3. Latest consensus state: current_term: 1 leader_uuid: "f10599e6b05a47138f811c6f4a5c87e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10599e6b05a47138f811c6f4a5c87e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42975 } } }
I20260620 22:02:51.551975 28264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:51.552445 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:51.550344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:34449 (local address 127.0.0.1:42975)
0620 22:02:51.550507 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:51.550516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:51.550726 (+ 210us) server_negotiation.cc:407] Connection header received
0620 22:02:51.550893 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:51.550900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:51.550930 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:51.551090 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:51.551846 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:51.551850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:51.551851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:51.551856 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:51.551908 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:51.551926 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:51.552155 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:51.552199 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 22:02:51.552243 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
W20260620 22:02:51.553194 28280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:02:51.553280 28280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:02:51.553339 28268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:51.553524 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:51.553561 28268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:51.555171 28268 catalog_manager.cc:1347] Generated new cluster ID: 27b3bb24f254485da71ba4677bfc48c9
I20260620 22:02:51.555226 28268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:51.556234 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:51.555028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:57099 (local address 127.0.0.1:42975)
0620 22:02:51.555061 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:51.555063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:51.555346 (+ 283us) server_negotiation.cc:407] Connection header received
0620 22:02:51.555374 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:51.555374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:51.555378 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:51.555433 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:51.556032 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:51.556033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:51.556033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:51.556037 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:51.556052 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:51.556056 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:51.556101 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:51.556162 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 22:02:51.556175 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:02:51.637923 28268 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:51.638129 28268 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:51.643787 28268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f10599e6b05a47138f811c6f4a5c87e3: Generated new TSK 0
I20260620 22:02:51.643973 28268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:51.651111 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 28191
2026-06-20T22:02:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:51 dist-test-slave-gl2g krb5kdc[28290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:51 dist-test-slave-gl2g krb5kdc[28290](info): set up 2 sockets
Jun 20 22:02:51 dist-test-slave-gl2g krb5kdc[28290](info): commencing operation
krb5kdc: starting...
W20260620 22:02:52.682829 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:52 dist-test-slave-gl2g krb5kdc[28290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:52.793612 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:52.872380 28303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:52.872582 28303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:52.872604 28303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:52.873668 28303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:52.873701 28303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:52.873715 28303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:52.873728 28303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:52.873739 28303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:52.875134 28303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:52.875315 28303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:52.875506 28303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:52.880261 28309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:52.880271 28303 server_base.cc:1034] running on GCE node
W20260620 22:02:52.880254 28311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:52.880252 28308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:52.880767 28303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:52.881042 28303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:52.882229 28303 hybrid_clock.cc:648] HybridClock initialized: now 1781992972882206 us; error 35 us; skew 500 ppm
Jun 20 22:02:52 dist-test-slave-gl2g krb5kdc[28290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:52.889536 28303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:52.890110 28303 webserver.cc:460] Webserver started at http://127.0.0.1:38425/ using document root <none> and password file <none>
I20260620 22:02:52.890406 28303 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:52.890467 28303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:52.890625 28303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:52.892174 28303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "73c04a3969264d0d994426a56d9f8c34"
format_stamp: "Formatted at 2026-06-20 22:02:52 on dist-test-slave-gl2g"
server_key: "24ef43eaf573e1f865cbbd78322c254d"
server_key_iv: "2dc149f39b7c9285b1e972edf33ddf27"
server_key_version: "encryptionkey@0"
I20260620 22:02:52.892596 28303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "73c04a3969264d0d994426a56d9f8c34"
format_stamp: "Formatted at 2026-06-20 22:02:52 on dist-test-slave-gl2g"
server_key: "24ef43eaf573e1f865cbbd78322c254d"
server_key_iv: "2dc149f39b7c9285b1e972edf33ddf27"
server_key_version: "encryptionkey@0"
I20260620 22:02:52.894384 28303 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 22:02:52.895380 28318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:52.895547 28303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:52.895678 28303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "73c04a3969264d0d994426a56d9f8c34"
format_stamp: "Formatted at 2026-06-20 22:02:52 on dist-test-slave-gl2g"
server_key: "24ef43eaf573e1f865cbbd78322c254d"
server_key_iv: "2dc149f39b7c9285b1e972edf33ddf27"
server_key_version: "encryptionkey@0"
I20260620 22:02:52.895759 28303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:52.920285 28303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:52.922979 28303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:52.923133 28303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:52.927609 28303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33617
I20260620 22:02:52.927657 28370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33617 every 8 connection(s)
I20260620 22:02:52.928032 28303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:52.928532 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 28303
I20260620 22:02:52.928653 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:52.928861 7110 external_mini_cluster.cc:1428] Setting key 0ec569c0df59cbd24fe1975218060f67
I20260620 22:02:52.929318 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:52.933293 28371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34: Bootstrap starting.
I20260620 22:02:52.934088 28371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:52.934418 28371 log.cc:824] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:52.935154 28371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34: No bootstrap required, opened a new log
Jun 20 22:02:52 dist-test-slave-gl2g krb5kdc[28290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:02:52.936944 28371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c04a3969264d0d994426a56d9f8c34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } }
I20260620 22:02:52.937100 28371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:52.937147 28371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c04a3969264d0d994426a56d9f8c34, State: Initialized, Role: FOLLOWER
I20260620 22:02:52.937295 28371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c04a3969264d0d994426a56d9f8c34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } }
I20260620 22:02:52.937397 28371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:52.937439 28371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:52.937489 28371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:52.937989 28371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c04a3969264d0d994426a56d9f8c34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } }
I20260620 22:02:52.938103 28371 leader_election.cc:302] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73c04a3969264d0d994426a56d9f8c34; no voters:
I20260620 22:02:52.938283 28371 leader_election.cc:288] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:52.938339 28376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:52.938503 28376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [term 1 LEADER]: Becoming Leader. State: Replica: 73c04a3969264d0d994426a56d9f8c34, State: Running, Role: LEADER
I20260620 22:02:52.938546 28371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:52.938582 28376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c04a3969264d0d994426a56d9f8c34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } }
I20260620 22:02:52.938982 28377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73c04a3969264d0d994426a56d9f8c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c04a3969264d0d994426a56d9f8c34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } } }
I20260620 22:02:52.938984 28378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73c04a3969264d0d994426a56d9f8c34. Latest consensus state: current_term: 1 leader_uuid: "73c04a3969264d0d994426a56d9f8c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c04a3969264d0d994426a56d9f8c34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } } }
I20260620 22:02:52.939124 28378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:52.939356 28385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:52.939548 28377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:52.939811 28385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:52.941264 28385 catalog_manager.cc:1347] Generated new cluster ID: 53e39850d0ed427aa37f8c0667339b9a
I20260620 22:02:52.941325 28385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:52.944775 28374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:52.929384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:52595 (local address 127.0.0.1:33617)
0620 22:02:52.929612 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:52.929629 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:52.929660 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:02:52.930109 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:52.930116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:52.930162 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:52.930310 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:52.930985 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:52.931868 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:52.932703 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:52.933122 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:52.933249 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:52.940428 (+ 7179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:52.940432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:52.940433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:52.940445 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:52.942919 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:52.943379 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:52.943382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:52.943382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:52.943499 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:52.943746 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:52.943747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:52.943747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:52.944033 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:52.944198 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:52.944333 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 22:02:52.944433 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260620 22:02:52.945755 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:52.954291 28374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:52.947234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:58891 (local address 127.0.0.1:33617)
0620 22:02:52.947269 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:52.947271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:52.947344 (+ 73us) server_negotiation.cc:407] Connection header received
0620 22:02:52.947400 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:52.947401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:52.947410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:52.947459 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:52.948085 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:52.948803 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:52.949798 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:52.950108 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:52.950169 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:52.951603 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:52.951604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:52.951604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:52.951611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:52.953090 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:52.953496 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:52.953497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:52.953497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:52.953550 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:52.953778 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:52.953779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:52.953779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:52.953899 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:52.953986 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:52.954145 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 22:02:52.954206 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:02:52.978472 28385 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:52.979092 28385 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:52.984570 28385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73c04a3969264d0d994426a56d9f8c34: Generated new TSK 0
I20260620 22:02:52.984767 28385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:52.992975 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 28303
W20260620 22:02:52.997547 28288 connection.cc:535] client connection to 127.0.0.1:33617 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33617): Connection reset by peer (error 104)
2026-06-20T22:02:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:53 dist-test-slave-gl2g krb5kdc[28402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:53 dist-test-slave-gl2g krb5kdc[28402](info): set up 2 sockets
Jun 20 22:02:53 dist-test-slave-gl2g krb5kdc[28402](info): commencing operation
krb5kdc: starting...
W20260620 22:02:54.034724 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:54 dist-test-slave-gl2g krb5kdc[28402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:54.152792 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:54.231544 28415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:54.231805 28415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:54.231842 28415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:54.233037 28415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:54.233116 28415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:54.233144 28415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:54.233174 28415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:54.233199 28415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:54.234795 28415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38685
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:54.235086 28415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:54.235323 28415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:54.240185 28423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:54.240211 28421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:54.240195 28420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:54.240450 28415 server_base.cc:1034] running on GCE node
I20260620 22:02:54.240739 28415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:54.241030 28415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:54.242209 28415 hybrid_clock.cc:648] HybridClock initialized: now 1781992974242202 us; error 35 us; skew 500 ppm
Jun 20 22:02:54 dist-test-slave-gl2g krb5kdc[28402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:54.244663 28415 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:54.245033 28415 webserver.cc:460] Webserver started at http://127.0.0.1:34861/ using document root <none> and password file <none>
I20260620 22:02:54.245215 28415 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:54.245254 28415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:54.245342 28415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:54.246336 28415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "f48d11eb1e91471bbcaf7db732833f2d"
format_stamp: "Formatted at 2026-06-20 22:02:54 on dist-test-slave-gl2g"
server_key: "f0d5528d7afabcd0fa9eaa24d5b0d966"
server_key_iv: "2720273bd98c25de29ec87b50a55f3db"
server_key_version: "encryptionkey@0"
I20260620 22:02:54.246702 28415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "f48d11eb1e91471bbcaf7db732833f2d"
format_stamp: "Formatted at 2026-06-20 22:02:54 on dist-test-slave-gl2g"
server_key: "f0d5528d7afabcd0fa9eaa24d5b0d966"
server_key_iv: "2720273bd98c25de29ec87b50a55f3db"
server_key_version: "encryptionkey@0"
I20260620 22:02:54.248003 28415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:54.248734 28430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:54.248900 28415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:54.248989 28415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "f48d11eb1e91471bbcaf7db732833f2d"
format_stamp: "Formatted at 2026-06-20 22:02:54 on dist-test-slave-gl2g"
server_key: "f0d5528d7afabcd0fa9eaa24d5b0d966"
server_key_iv: "2720273bd98c25de29ec87b50a55f3db"
server_key_version: "encryptionkey@0"
I20260620 22:02:54.249045 28415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:54.257468 28415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:54.259512 28415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:54.259631 28415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:54.263550 28415 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38685
I20260620 22:02:54.263612 28482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38685 every 8 connection(s)
I20260620 22:02:54.263965 28415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:54.264834 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:54.267230 28483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d: Bootstrap starting.
I20260620 22:02:54.267239 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 28415
I20260620 22:02:54.267350 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:54.267541 7110 external_mini_cluster.cc:1428] Setting key daff78a750d096fad0b4800eff9af34c
I20260620 22:02:54.267922 28483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:54.268278 28483 log.cc:824] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:54.268956 28483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d: No bootstrap required, opened a new log
I20260620 22:02:54.270762 28483 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d11eb1e91471bbcaf7db732833f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260620 22:02:54.270915 28483 raft_consensus.cc:374] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:54.270957 28483 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48d11eb1e91471bbcaf7db732833f2d, State: Initialized, Role: FOLLOWER
I20260620 22:02:54.271111 28483 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d11eb1e91471bbcaf7db732833f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260620 22:02:54.271204 28483 raft_consensus.cc:388] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:54.271250 28483 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:54.271301 28483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:54.271818 28483 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d11eb1e91471bbcaf7db732833f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260620 22:02:54.271934 28483 leader_election.cc:302] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f48d11eb1e91471bbcaf7db732833f2d; no voters:
I20260620 22:02:54.272111 28483 leader_election.cc:288] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:54.272238 28488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:54.272359 28483 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:54.272398 28488 raft_consensus.cc:686] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [term 1 LEADER]: Becoming Leader. State: Replica: f48d11eb1e91471bbcaf7db732833f2d, State: Running, Role: LEADER
I20260620 22:02:54.272472 28488 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d11eb1e91471bbcaf7db732833f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260620 22:02:54.272825 28489 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f48d11eb1e91471bbcaf7db732833f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d11eb1e91471bbcaf7db732833f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } } }
I20260620 22:02:54.272912 28489 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:54.273061 28490 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48d11eb1e91471bbcaf7db732833f2d. Latest consensus state: current_term: 1 leader_uuid: "f48d11eb1e91471bbcaf7db732833f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d11eb1e91471bbcaf7db732833f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } } }
I20260620 22:02:54.273118 28490 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:54.273407 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:54.268088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:44381 (local address 127.0.0.1:38685)
0620 22:02:54.268308 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:54.268316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:54.268340 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:02:54.268816 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:54.268823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:54.268870 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:54.269021 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:54.269215 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:54.270084 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:54.270996 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:54.271515 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:54.271663 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:54.272441 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:54.272445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:54.272445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:54.272658 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:54.272695 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:54.272709 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:54.272769 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:54.273157 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 22:02:54.273192 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260620 22:02:54.273964 28496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:54.274413 28496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:54.274652 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:54.276078 28496 catalog_manager.cc:1347] Generated new cluster ID: 65fca1222528444692077a9758885a1b
I20260620 22:02:54.276131 28496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:54.281584 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:54.276095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:38611 (local address 127.0.0.1:38685)
0620 22:02:54.276125 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:54.276127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:54.278925 (+ 2798us) server_negotiation.cc:407] Connection header received
0620 22:02:54.279002 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:54.279003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:54.279014 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:54.279060 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:54.279207 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:54.279676 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:54.280380 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:54.280599 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:54.280685 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:54.281342 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:54.281344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:54.281344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:54.281349 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:54.281362 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:54.281366 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:54.281413 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:54.281482 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 22:02:54.281491 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 22:02:54.298342 28496 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:54.298939 28496 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:54.305442 28496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f48d11eb1e91471bbcaf7db732833f2d: Generated new TSK 0
I20260620 22:02:54.305655 28496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:54.314086 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 28415
W20260620 22:02:54.318428 28400 connection.cc:535] client connection to 127.0.0.1:38685 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38685): Connection reset by peer (error 104)
2026-06-20T22:02:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:54 dist-test-slave-gl2g krb5kdc[28514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:54 dist-test-slave-gl2g krb5kdc[28514](info): set up 2 sockets
Jun 20 22:02:54 dist-test-slave-gl2g krb5kdc[28514](info): commencing operation
krb5kdc: starting...
W20260620 22:02:55.339200 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:55 dist-test-slave-gl2g krb5kdc[28514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:02:55.449524 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:55.528637 28527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:55.528825 28527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:55.528863 28527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:55.529919 28527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:55.529953 28527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:55.529966 28527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:55.529978 28527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:55.529990 28527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:55.531337 28527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:55.531540 28527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:55.531698 28527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:55.536478 28532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:55.536487 28533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:55.536477 28535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:55.536772 28527 server_base.cc:1034] running on GCE node
I20260620 22:02:55.537004 28527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:55.537307 28527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:55.538470 28527 hybrid_clock.cc:648] HybridClock initialized: now 1781992975538467 us; error 63 us; skew 500 ppm
Jun 20 22:02:55 dist-test-slave-gl2g krb5kdc[28514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:55.540932 28527 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:02:55.541383 28527 webserver.cc:460] Webserver started at http://127.0.0.1:42025/ using document root <none> and password file <none>
I20260620 22:02:55.541613 28527 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:55.541680 28527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:55.541826 28527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:55.542935 28527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7"
format_stamp: "Formatted at 2026-06-20 22:02:55 on dist-test-slave-gl2g"
server_key: "d6ca2bea23acad8372826929fca4e442"
server_key_iv: "31dc5379954646f93f08220f2af669c4"
server_key_version: "encryptionkey@0"
I20260620 22:02:55.543282 28527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7"
format_stamp: "Formatted at 2026-06-20 22:02:55 on dist-test-slave-gl2g"
server_key: "d6ca2bea23acad8372826929fca4e442"
server_key_iv: "31dc5379954646f93f08220f2af669c4"
server_key_version: "encryptionkey@0"
I20260620 22:02:55.544584 28527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:02:55.545327 28542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:55.545533 28527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:02:55.545631 28527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7"
format_stamp: "Formatted at 2026-06-20 22:02:55 on dist-test-slave-gl2g"
server_key: "d6ca2bea23acad8372826929fca4e442"
server_key_iv: "31dc5379954646f93f08220f2af669c4"
server_key_version: "encryptionkey@0"
I20260620 22:02:55.545710 28527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:55.548918 28527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:55.551184 28527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:55.551334 28527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:55.555224 28527 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42665
I20260620 22:02:55.555258 28594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42665 every 8 connection(s)
I20260620 22:02:55.555572 28527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:55.556478 28595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:55.558696 28595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7: Bootstrap starting.
I20260620 22:02:55.559266 28595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:55.559546 28595 log.cc:824] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:55.560209 28595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7: No bootstrap required, opened a new log
I20260620 22:02:55.561708 28595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42665 } }
I20260620 22:02:55.561837 28595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:55.561883 28595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3713b9bf14da4a0ea5b1da58e8e4fde7, State: Initialized, Role: FOLLOWER
I20260620 22:02:55.562052 28595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42665 } }
I20260620 22:02:55.562114 28595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:55.562135 28595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:55.562196 28595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:55.562822 28595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42665 } }
I20260620 22:02:55.562934 28595 leader_election.cc:302] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3713b9bf14da4a0ea5b1da58e8e4fde7; no voters:
I20260620 22:02:55.563100 28595 leader_election.cc:288] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:55.563153 28598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:55.563330 28598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [term 1 LEADER]: Becoming Leader. State: Replica: 3713b9bf14da4a0ea5b1da58e8e4fde7, State: Running, Role: LEADER
I20260620 22:02:55.563350 28595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:55.563431 28598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42665 } }
I20260620 22:02:55.563831 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 28527
I20260620 22:02:55.563948 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:55.564042 28599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42665 } } }
I20260620 22:02:55.564047 28600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3713b9bf14da4a0ea5b1da58e8e4fde7. Latest consensus state: current_term: 1 leader_uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3713b9bf14da4a0ea5b1da58e8e4fde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42665 } } }
I20260620 22:02:55.564144 7110 external_mini_cluster.cc:1428] Setting key fce001c0098687a958a84303d68ece68
I20260620 22:02:55.564126 28599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:55.564129 28600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:55.564324 28607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:55.564792 28607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:55.566390 28607 catalog_manager.cc:1347] Generated new cluster ID: a5dbc979d2574bdab83f23d9422cd4a1
I20260620 22:02:55.566447 28607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:55.566891 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:55.564756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:38935 (local address 127.0.0.1:42665)
0620 22:02:55.564921 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:55.564930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:55.565156 (+ 226us) server_negotiation.cc:407] Connection header received
0620 22:02:55.565365 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:55.565373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:55.565408 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:55.565558 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:55.566371 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:55.566374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:55.566374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:55.566378 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:55.566423 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:55.566439 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:55.566660 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:55.566692 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 22:02:55.566732 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260620 22:02:55.567745 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:55.571741 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:55.569158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:49475 (local address 127.0.0.1:42665)
0620 22:02:55.569188 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:55.569190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:55.570782 (+ 1592us) server_negotiation.cc:407] Connection header received
0620 22:02:55.570879 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:55.570880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:55.570886 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:55.570946 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:55.571525 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:55.571526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:55.571526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:55.571530 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:55.571545 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:55.571551 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:55.571613 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:55.571665 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 22:02:55.571678 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:02:55.575610 28607 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:55.575768 28607 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:55.580929 28607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3713b9bf14da4a0ea5b1da58e8e4fde7: Generated new TSK 0
I20260620 22:02:55.581130 28607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:55.583730 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 28527
2026-06-20T22:02:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:55 dist-test-slave-gl2g krb5kdc[28626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:55 dist-test-slave-gl2g krb5kdc[28626](info): set up 2 sockets
Jun 20 22:02:55 dist-test-slave-gl2g krb5kdc[28626](info): commencing operation
krb5kdc: starting...
W20260620 22:02:56.638774 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:56 dist-test-slave-gl2g krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:02:56.751245 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:45893
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:45893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:56.832852 28639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:56.833055 28639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:56.833077 28639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:56.834223 28639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:56.834264 28639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:56.834278 28639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:56.834291 28639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:56.834342 28639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:56.835922 28639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:45893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:45893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:56.836179 28639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:56.836408 28639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:02:56.841275 28639 server_base.cc:1034] running on GCE node
W20260620 22:02:56.841260 28647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:56.841241 28644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:56.841235 28645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:56.841713 28639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:56.842000 28639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:56.843164 28639 hybrid_clock.cc:648] HybridClock initialized: now 1781992976843156 us; error 37 us; skew 500 ppm
Jun 20 22:02:56 dist-test-slave-gl2g krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992976, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:56.845567 28639 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:02:56.846029 28639 webserver.cc:460] Webserver started at http://127.6.241.190:36721/ using document root <none> and password file <none>
I20260620 22:02:56.846273 28639 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:56.846340 28639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:56.846487 28639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:56.847589 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "4c7449cfa35b4c52a21173a8ae7c7484"
format_stamp: "Formatted at 2026-06-20 22:02:56 on dist-test-slave-gl2g"
server_key: "c0aa9a8c0ff7e0667ae9504297c05e81"
server_key_iv: "53e5224a9559634fd3b6059e53b76ac6"
server_key_version: "encryptionkey@0"
I20260620 22:02:56.847955 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "4c7449cfa35b4c52a21173a8ae7c7484"
format_stamp: "Formatted at 2026-06-20 22:02:56 on dist-test-slave-gl2g"
server_key: "c0aa9a8c0ff7e0667ae9504297c05e81"
server_key_iv: "53e5224a9559634fd3b6059e53b76ac6"
server_key_version: "encryptionkey@0"
I20260620 22:02:56.849249 28639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:02:56.849946 28654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:56.850108 28639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:02:56.850212 28639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "4c7449cfa35b4c52a21173a8ae7c7484"
format_stamp: "Formatted at 2026-06-20 22:02:56 on dist-test-slave-gl2g"
server_key: "c0aa9a8c0ff7e0667ae9504297c05e81"
server_key_iv: "53e5224a9559634fd3b6059e53b76ac6"
server_key_version: "encryptionkey@0"
I20260620 22:02:56.850294 28639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:56.876756 28639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:56.878832 28639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:56.878993 28639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:56.882865 28639 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:45893
I20260620 22:02:56.882907 28706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:45893 every 8 connection(s)
I20260620 22:02:56.883296 28639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:56.884174 28707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:56.885829 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 28639
I20260620 22:02:56.885953 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:56.886132 7110 external_mini_cluster.cc:1428] Setting key ea80b0a625ddca4c50c37a68bdea74ab
I20260620 22:02:56.886746 28707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484: Bootstrap starting.
I20260620 22:02:56.887384 28707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:56.887701 28707 log.cc:824] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:56.888391 28707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484: No bootstrap required, opened a new log
I20260620 22:02:56.890183 28707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7449cfa35b4c52a21173a8ae7c7484" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45893 } }
I20260620 22:02:56.890340 28707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:56.890386 28707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7449cfa35b4c52a21173a8ae7c7484, State: Initialized, Role: FOLLOWER
I20260620 22:02:56.890571 28707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7449cfa35b4c52a21173a8ae7c7484" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45893 } }
I20260620 22:02:56.890713 28707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:56.890772 28707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:56.890826 28707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:56.891518 28707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7449cfa35b4c52a21173a8ae7c7484" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45893 } }
I20260620 22:02:56.891606 28707 leader_election.cc:302] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c7449cfa35b4c52a21173a8ae7c7484; no voters:
I20260620 22:02:56.891790 28707 leader_election.cc:288] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:56.891808 28712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:56.891922 28712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [term 1 LEADER]: Becoming Leader. State: Replica: 4c7449cfa35b4c52a21173a8ae7c7484, State: Running, Role: LEADER
I20260620 22:02:56.892068 28712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7449cfa35b4c52a21173a8ae7c7484" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45893 } }
I20260620 22:02:56.892066 28707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 22:02:56 dist-test-slave-gl2g krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:02:56.893289 28714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c7449cfa35b4c52a21173a8ae7c7484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7449cfa35b4c52a21173a8ae7c7484" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45893 } } }
I20260620 22:02:56.893391 28714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:56.893615 28727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:56.893640 28713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c7449cfa35b4c52a21173a8ae7c7484. Latest consensus state: current_term: 1 leader_uuid: "4c7449cfa35b4c52a21173a8ae7c7484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7449cfa35b4c52a21173a8ae7c7484" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 45893 } } }
I20260620 22:02:56.893807 28713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:56.894227 28727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:56.896291 28727 catalog_manager.cc:1347] Generated new cluster ID: f3377d1a290a41eab01f348aac40dbcb
I20260620 22:02:56.896344 28727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:56.897337 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:56.886718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49931 (local address 127.6.241.190:45893)
0620 22:02:56.886882 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:56.886890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:56.886915 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:02:56.887340 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:56.887347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:56.887394 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:56.887551 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:56.888335 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:56.889257 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:56.890025 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:56.890494 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:56.890628 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:56.893421 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:56.893426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:56.893426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:56.893439 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:56.895534 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:56.895948 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:56.895951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:56.895952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:56.896058 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:56.896378 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:56.896379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:56.896380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:56.896694 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:56.896855 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:56.897007 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 22:02:56.897120 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260620 22:02:56.898125 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:56.906852 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:56.899553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52833 (local address 127.6.241.190:45893)
0620 22:02:56.899598 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:56.899601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:56.899633 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:02:56.899678 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:56.899680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:56.899689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:56.899751 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:56.900368 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:56.901142 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:56.902144 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:56.902479 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:56.902555 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:56.903966 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:56.903967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:56.903968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:02:56.903976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:56.905589 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:56.905994 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:56.905995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:56.905995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:56.906057 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:02:56.906278 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:02:56.906279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:02:56.906279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:02:56.906410 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:56.906526 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:56.906692 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 22:02:56.906756 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:02:56.932653 28727 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:56.933409 28727 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:56.942440 28727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c7449cfa35b4c52a21173a8ae7c7484: Generated new TSK 0
I20260620 22:02:56.942708 28727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:56.956277 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 28639
W20260620 22:02:56.960662 28624 connection.cc:535] client connection to 127.6.241.190:45893 recv error: Network error: failed to read from TLS socket (remote: 127.6.241.190:45893): Connection reset by peer (error 104)
2026-06-20T22:02:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:57 dist-test-slave-gl2g krb5kdc[28738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:57 dist-test-slave-gl2g krb5kdc[28738](info): set up 2 sockets
Jun 20 22:02:57 dist-test-slave-gl2g krb5kdc[28738](info): commencing operation
krb5kdc: starting...
W20260620 22:02:57.986828 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:58 dist-test-slave-gl2g krb5kdc[28738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:02:58.099371 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:38291
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:38223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:38291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:58.181324 28751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:58.181582 28751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:58.181622 28751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:58.182763 28751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:58.182834 28751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:58.182858 28751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:58.182888 28751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:58.182911 28751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:58.184357 28751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:38223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:38291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:38291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:58.184643 28751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:58.184863 28751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:58.189667 28757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:58.189632 28759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:58.189693 28756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:58.189739 28751 server_base.cc:1034] running on GCE node
I20260620 22:02:58.190058 28751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:58.190354 28751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:58.191515 28751 hybrid_clock.cc:648] HybridClock initialized: now 1781992978191501 us; error 42 us; skew 500 ppm
Jun 20 22:02:58 dist-test-slave-gl2g krb5kdc[28738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992978, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:58.197381 28751 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:02:58.197892 28751 webserver.cc:460] Webserver started at http://127.6.241.190:41211/ using document root <none> and password file <none>
I20260620 22:02:58.198117 28751 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:58.198160 28751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:58.198298 28751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:58.199394 28751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "b9b234f35dd44914b571e934bad765a5"
format_stamp: "Formatted at 2026-06-20 22:02:58 on dist-test-slave-gl2g"
server_key: "bef13053081cd57926d010508f4c607a"
server_key_iv: "a5bf395471c62993052239ea8b436a51"
server_key_version: "encryptionkey@0"
I20260620 22:02:58.199760 28751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "b9b234f35dd44914b571e934bad765a5"
format_stamp: "Formatted at 2026-06-20 22:02:58 on dist-test-slave-gl2g"
server_key: "bef13053081cd57926d010508f4c607a"
server_key_iv: "a5bf395471c62993052239ea8b436a51"
server_key_version: "encryptionkey@0"
I20260620 22:02:58.201053 28751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:02:58.201865 28766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:58.202049 28751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:02:58.202157 28751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "b9b234f35dd44914b571e934bad765a5"
format_stamp: "Formatted at 2026-06-20 22:02:58 on dist-test-slave-gl2g"
server_key: "bef13053081cd57926d010508f4c607a"
server_key_iv: "a5bf395471c62993052239ea8b436a51"
server_key_version: "encryptionkey@0"
I20260620 22:02:58.202230 28751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:58.220461 28751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:58.222420 28751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:58.222576 28751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:58.226328 28751 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:38291
I20260620 22:02:58.226397 28818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:38291 every 8 connection(s)
I20260620 22:02:58.226795 28751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:58.227738 28819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:58.230002 28819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5: Bootstrap starting.
I20260620 22:02:58.230612 28819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:58.230903 28819 log.cc:824] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:58.231597 28819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5: No bootstrap required, opened a new log
I20260620 22:02:58.233167 28819 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b234f35dd44914b571e934bad765a5" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38291 } }
I20260620 22:02:58.233304 28819 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:58.233325 28819 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b234f35dd44914b571e934bad765a5, State: Initialized, Role: FOLLOWER
I20260620 22:02:58.233522 28819 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b234f35dd44914b571e934bad765a5" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38291 } }
I20260620 22:02:58.233585 28819 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:58.233608 28819 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:58.233675 28819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:58.233837 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 28751
I20260620 22:02:58.233955 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:58.234143 7110 external_mini_cluster.cc:1428] Setting key 94db1a792236ff530cfa3a7aa5664a50
I20260620 22:02:58.234825 28819 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b234f35dd44914b571e934bad765a5" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38291 } }
I20260620 22:02:58.234936 28819 leader_election.cc:302] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9b234f35dd44914b571e934bad765a5; no voters:
I20260620 22:02:58.235085 28819 leader_election.cc:288] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:58.235133 28824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:58.235278 28824 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [term 1 LEADER]: Becoming Leader. State: Replica: b9b234f35dd44914b571e934bad765a5, State: Running, Role: LEADER
I20260620 22:02:58.235333 28819 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:58.235397 28824 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b234f35dd44914b571e934bad765a5" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38291 } }
I20260620 22:02:58.235811 28824 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9b234f35dd44914b571e934bad765a5. Latest consensus state: current_term: 1 leader_uuid: "b9b234f35dd44914b571e934bad765a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b234f35dd44914b571e934bad765a5" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38291 } } }
I20260620 22:02:58.235821 28825 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9b234f35dd44914b571e934bad765a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b234f35dd44914b571e934bad765a5" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38291 } } }
I20260620 22:02:58.235965 28824 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:58.236022 28825 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:58.236460 28831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:58.237005 28831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:58.239682 28831 catalog_manager.cc:1347] Generated new cluster ID: 95d367233b574ada91a56b75380bd3bf
I20260620 22:02:58.239738 28831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:58.240192 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:58.234666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39075 (local address 127.6.241.190:38291)
0620 22:02:58.234886 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:58.234896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:58.234941 (+ 45us) server_negotiation.cc:407] Connection header received
0620 22:02:58.235071 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:58.235079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:58.235136 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:58.235576 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:58.235667 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:58.237254 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:58.238054 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:58.238540 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:58.238695 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:58.239440 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:58.239443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:58.239444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:58.239646 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:58.239684 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:58.239699 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:58.239780 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:58.239964 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 22:02:58.239989 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260620 22:02:58.243619 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:58.254490 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:58.246919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51155 (local address 127.6.241.190:38291)
0620 22:02:58.246971 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:58.246973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:58.250804 (+ 3831us) server_negotiation.cc:407] Connection header received
0620 22:02:58.250872 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:58.250873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:58.250885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:58.250949 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:58.251126 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:58.251942 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:58.253055 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:02:58.253372 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:02:58.253430 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:02:58.254211 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:58.254213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:58.254213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:58.254218 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:58.254232 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:58.254236 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:58.254289 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:58.254387 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 22:02:58.254400 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:02:58.259832 28831 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:58.260329 28831 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:58.267690 28831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9b234f35dd44914b571e934bad765a5: Generated new TSK 0
I20260620 22:02:58.267858 28831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:58.277952 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 28751
W20260620 22:02:58.282327 28736 connection.cc:535] client connection to 127.6.241.190:38291 recv error: Network error: failed to read from TLS socket (remote: 127.6.241.190:38291): Connection reset by peer (error 104)
2026-06-20T22:02:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:58 dist-test-slave-gl2g krb5kdc[28850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:58 dist-test-slave-gl2g krb5kdc[28850](info): set up 2 sockets
Jun 20 22:02:58 dist-test-slave-gl2g krb5kdc[28850](info): commencing operation
krb5kdc: starting...
W20260620 22:02:59.310730 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:02:59 dist-test-slave-gl2g krb5kdc[28850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:02:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:02:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:02:59.420374 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42205
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:42205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:02:59.499864 28863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:02:59.500087 28863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:02:59.500110 28863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:02:59.501189 28863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:02:59.501222 28863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:02:59.501236 28863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:02:59.501248 28863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:02:59.501260 28863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:02:59.502696 28863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:42205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:02:59.502873 28863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:02:59.503031 28863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:02:59.507863 28871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:59.507879 28868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:02:59.507879 28869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:02:59.507958 28863 server_base.cc:1034] running on GCE node
I20260620 22:02:59.508513 28863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:02:59.508800 28863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:02:59.509958 28863 hybrid_clock.cc:648] HybridClock initialized: now 1781992979509951 us; error 38 us; skew 500 ppm
Jun 20 22:02:59 dist-test-slave-gl2g krb5kdc[28850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:02:59.512373 28863 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:02:59.512746 28863 webserver.cc:460] Webserver started at http://127.6.241.190:38185/ using document root <none> and password file <none>
I20260620 22:02:59.512933 28863 fs_manager.cc:359] Metadata directory not provided
I20260620 22:02:59.512972 28863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:02:59.513057 28863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:02:59.514103 28863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "2c724cfc098e4c8b97fbe44d40338eb8"
format_stamp: "Formatted at 2026-06-20 22:02:59 on dist-test-slave-gl2g"
server_key: "40cf96a557d0cbc3d1dd2bb14942e990"
server_key_iv: "459c2e3c5043e7bfb724cc86e2b61bb8"
server_key_version: "encryptionkey@0"
I20260620 22:02:59.514392 28863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "2c724cfc098e4c8b97fbe44d40338eb8"
format_stamp: "Formatted at 2026-06-20 22:02:59 on dist-test-slave-gl2g"
server_key: "40cf96a557d0cbc3d1dd2bb14942e990"
server_key_iv: "459c2e3c5043e7bfb724cc86e2b61bb8"
server_key_version: "encryptionkey@0"
I20260620 22:02:59.515712 28863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:02:59.516367 28878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:02:59.516516 28863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:02:59.516609 28863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "2c724cfc098e4c8b97fbe44d40338eb8"
format_stamp: "Formatted at 2026-06-20 22:02:59 on dist-test-slave-gl2g"
server_key: "40cf96a557d0cbc3d1dd2bb14942e990"
server_key_iv: "459c2e3c5043e7bfb724cc86e2b61bb8"
server_key_version: "encryptionkey@0"
I20260620 22:02:59.516664 28863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:02:59.519894 28863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:02:59.522024 28863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:02:59.522130 28863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:02:59.525815 28863 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:42205
I20260620 22:02:59.525859 28930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:42205 every 8 connection(s)
I20260620 22:02:59.526150 28863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:02:59.527151 28931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:02:59.529290 28931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8: Bootstrap starting.
I20260620 22:02:59.529861 28931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8: Neither blocks nor log segments found. Creating new log.
I20260620 22:02:59.530139 28931 log.cc:824] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:02:59.530913 28931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8: No bootstrap required, opened a new log
I20260620 22:02:59.532455 28931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c724cfc098e4c8b97fbe44d40338eb8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42205 } }
I20260620 22:02:59.532614 28931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:02:59.532661 28931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c724cfc098e4c8b97fbe44d40338eb8, State: Initialized, Role: FOLLOWER
I20260620 22:02:59.532850 28931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c724cfc098e4c8b97fbe44d40338eb8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42205 } }
I20260620 22:02:59.532951 28931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:02:59.532994 28931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:02:59.533046 28931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:02:59.533903 28931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c724cfc098e4c8b97fbe44d40338eb8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42205 } }
I20260620 22:02:59.534024 28931 leader_election.cc:302] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c724cfc098e4c8b97fbe44d40338eb8; no voters:
I20260620 22:02:59.534212 28931 leader_election.cc:288] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:02:59.534253 28934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:02:59.534435 28934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [term 1 LEADER]: Becoming Leader. State: Replica: 2c724cfc098e4c8b97fbe44d40338eb8, State: Running, Role: LEADER
I20260620 22:02:59.534513 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 28863
I20260620 22:02:59.534469 28931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:02:59.534687 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:02:59.534636 28934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c724cfc098e4c8b97fbe44d40338eb8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42205 } }
I20260620 22:02:59.534883 7110 external_mini_cluster.cc:1428] Setting key 6ae5bc8f7dfae1e9fbf7019b6368c3ba
I20260620 22:02:59.535404 28935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c724cfc098e4c8b97fbe44d40338eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c724cfc098e4c8b97fbe44d40338eb8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42205 } } }
I20260620 22:02:59.535485 28935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:59.535647 28936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c724cfc098e4c8b97fbe44d40338eb8. Latest consensus state: current_term: 1 leader_uuid: "2c724cfc098e4c8b97fbe44d40338eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c724cfc098e4c8b97fbe44d40338eb8" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42205 } } }
I20260620 22:02:59.535719 28936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:02:59.535753 28945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:02:59.536351 28945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:02:59.537961 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:59.535586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47175 (local address 127.6.241.190:42205)
0620 22:02:59.536270 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:59.536281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:59.536315 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:02:59.536436 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:59.536444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:59.536478 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:59.536653 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:59.537405 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:59.537409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:59.537409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:59.537414 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:59.537469 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:59.537484 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:59.537618 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:59.537729 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 22:02:59.537771 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":73,"threads_started":1}
I20260620 22:02:59.538111 28945 catalog_manager.cc:1347] Generated new cluster ID: 944ea1e19df04b8ca827e1dec2076bae
I20260620 22:02:59.538159 28945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:02:59.538947 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:02:59.541463 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:02:59.540370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52509 (local address 127.6.241.190:42205)
0620 22:02:59.540412 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:02:59.540413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:02:59.540558 (+ 145us) server_negotiation.cc:407] Connection header received
0620 22:02:59.540605 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:02:59.540606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:02:59.540611 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:02:59.540685 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:02:59.541241 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:02:59.541243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:02:59.541243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:02:59.541247 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:02:59.541262 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:02:59.541266 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:02:59.541327 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:02:59.541378 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:02:59.541392 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:02:59.560585 28945 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:02:59.560774 28945 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:02:59.565999 28945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c724cfc098e4c8b97fbe44d40338eb8: Generated new TSK 0
I20260620 22:02:59.566195 28945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:02:59.571597 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 28863
2026-06-20T22:02:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:02:59 dist-test-slave-gl2g krb5kdc[28962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:02:59 dist-test-slave-gl2g krb5kdc[28962](info): set up 2 sockets
Jun 20 22:02:59 dist-test-slave-gl2g krb5kdc[28962](info): commencing operation
krb5kdc: starting...
W20260620 22:03:00.606848 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:03:00 dist-test-slave-gl2g krb5kdc[28962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:03:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:03:00.716713 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:39067
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:39067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:03:00.795473 28975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:03:00.795696 28975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:03:00.795733 28975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:03:00.796861 28975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:03:00.796929 28975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:03:00.796963 28975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:03:00.796984 28975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:03:00.797013 28975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:03:00.798462 28975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:39067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:39067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:03:00.798753 28975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:03:00.798969 28975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:03:00.803573 28980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:00.803562 28981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:00.803565 28983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:00.803623 28975 server_base.cc:1034] running on GCE node
I20260620 22:03:00.803977 28975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:03:00.804270 28975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:03:00.805405 28975 hybrid_clock.cc:648] HybridClock initialized: now 1781992980805389 us; error 29 us; skew 500 ppm
Jun 20 22:03:00 dist-test-slave-gl2g krb5kdc[28962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992980, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:03:00.808120 28975 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:03:00.808486 28975 webserver.cc:460] Webserver started at http://127.6.241.190:43667/ using document root <none> and password file <none>
I20260620 22:03:00.808677 28975 fs_manager.cc:359] Metadata directory not provided
I20260620 22:03:00.808727 28975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:03:00.808817 28975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:03:00.809821 28975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "6b7f0818e5774f1bbb2d816386c4ee9c"
format_stamp: "Formatted at 2026-06-20 22:03:00 on dist-test-slave-gl2g"
server_key: "b7cdb2866289a928ecd8418782bc79d4"
server_key_iv: "1ffa4dad8229bf8295f4c41800283439"
server_key_version: "encryptionkey@0"
I20260620 22:03:00.810117 28975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "6b7f0818e5774f1bbb2d816386c4ee9c"
format_stamp: "Formatted at 2026-06-20 22:03:00 on dist-test-slave-gl2g"
server_key: "b7cdb2866289a928ecd8418782bc79d4"
server_key_iv: "1ffa4dad8229bf8295f4c41800283439"
server_key_version: "encryptionkey@0"
I20260620 22:03:00.811508 28975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:03:00.812189 28990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:00.812386 28975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:03:00.812466 28975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "6b7f0818e5774f1bbb2d816386c4ee9c"
format_stamp: "Formatted at 2026-06-20 22:03:00 on dist-test-slave-gl2g"
server_key: "b7cdb2866289a928ecd8418782bc79d4"
server_key_iv: "1ffa4dad8229bf8295f4c41800283439"
server_key_version: "encryptionkey@0"
I20260620 22:03:00.812522 28975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:03:00.825088 28975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:03:00.826921 28975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:03:00.827029 28975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:03:00.830701 28975 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:39067
I20260620 22:03:00.830771 29042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:39067 every 8 connection(s)
I20260620 22:03:00.831087 28975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:03:00.831939 29043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:03:00.834168 29043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c: Bootstrap starting.
I20260620 22:03:00.834772 29043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c: Neither blocks nor log segments found. Creating new log.
I20260620 22:03:00.835057 29043 log.cc:824] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:03:00.835737 29043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c: No bootstrap required, opened a new log
I20260620 22:03:00.837278 29043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f0818e5774f1bbb2d816386c4ee9c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39067 } }
I20260620 22:03:00.837452 29043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:03:00.837502 29043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7f0818e5774f1bbb2d816386c4ee9c, State: Initialized, Role: FOLLOWER
I20260620 22:03:00.837666 29043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f0818e5774f1bbb2d816386c4ee9c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39067 } }
I20260620 22:03:00.837761 29043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:03:00.837805 29043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:03:00.837858 29043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:03:00.838682 29043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f0818e5774f1bbb2d816386c4ee9c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39067 } }
I20260620 22:03:00.838804 29043 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7f0818e5774f1bbb2d816386c4ee9c; no voters:
I20260620 22:03:00.838972 29043 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:03:00.838999 29046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:03:00.839164 29046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [term 1 LEADER]: Becoming Leader. State: Replica: 6b7f0818e5774f1bbb2d816386c4ee9c, State: Running, Role: LEADER
I20260620 22:03:00.839226 29043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:03:00.839303 29046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f0818e5774f1bbb2d816386c4ee9c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39067 } }
I20260620 22:03:00.839658 29048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7f0818e5774f1bbb2d816386c4ee9c. Latest consensus state: current_term: 1 leader_uuid: "6b7f0818e5774f1bbb2d816386c4ee9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f0818e5774f1bbb2d816386c4ee9c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39067 } } }
I20260620 22:03:00.839751 29048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:00.839653 29047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b7f0818e5774f1bbb2d816386c4ee9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f0818e5774f1bbb2d816386c4ee9c" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 39067 } } }
I20260620 22:03:00.839886 29047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:00.839989 29055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:03:00.840549 29055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:03:00.840869 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 28975
I20260620 22:03:00.840982 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:03:00.841153 7110 external_mini_cluster.cc:1428] Setting key 9de798ac48a38302c6f26bada89653fe
I20260620 22:03:00.842963 29055 catalog_manager.cc:1347] Generated new cluster ID: 026d7ac6165e49f2989195d57e958eee
I20260620 22:03:00.843024 29055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:03:00 dist-test-slave-gl2g krb5kdc[28962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:03:00.854769 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:00.841566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56197 (local address 127.6.241.190:39067)
0620 22:03:00.841731 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:00.841740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:00.841939 (+ 199us) server_negotiation.cc:407] Connection header received
0620 22:03:00.842061 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:00.842068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:00.842125 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:00.842283 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:00.846844 (+ 4561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:00.847765 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:00.848530 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:00.848964 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:00.849082 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:00.851435 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:00.851439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:00.851439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:00.851450 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:00.853096 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:00.853550 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:00.853553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:00.853554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:00.853646 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:00.853871 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:00.853872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:00.853872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:00.854115 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:00.854255 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:00.854377 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 22:03:00.854519 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260620 22:03:00.855578 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:03:00.862278 29055 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:03:00.863003 29055 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:03:00.868230 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:00.862045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57401 (local address 127.6.241.190:39067)
0620 22:03:00.862077 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:00.862078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:00.862260 (+ 182us) server_negotiation.cc:407] Connection header received
0620 22:03:00.862331 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:00.862332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:00.862343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:00.862389 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:00.862968 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:00.863497 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:00.864215 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:00.864427 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:00.864511 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:00.865701 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:00.865703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:00.865703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:00.865710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:00.867091 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:00.867453 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:00.867455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:00.867455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:00.867513 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:00.867709 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:00.867710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:00.867710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:00.867822 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:00.867947 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:00.868041 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 22:03:00.868132 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:03:00.869349 29055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b7f0818e5774f1bbb2d816386c4ee9c: Generated new TSK 0
I20260620 22:03:00.869518 29055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:03:00.876494 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 28975
2026-06-20T22:03:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:03:00 dist-test-slave-gl2g krb5kdc[29074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:03:00 dist-test-slave-gl2g krb5kdc[29074](info): set up 2 sockets
Jun 20 22:03:00 dist-test-slave-gl2g krb5kdc[29074](info): commencing operation
krb5kdc: starting...
W20260620 22:03:01.902820 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:03:01 dist-test-slave-gl2g krb5kdc[29074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:03:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:03:02.015609 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34687
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:03:02.094292 29087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:03:02.094489 29087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:03:02.094518 29087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:03:02.095723 29087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:03:02.095767 29087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:03:02.095781 29087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:03:02.095844 29087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:03:02.095870 29087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:03:02.097298 29087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:34687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:03:02.097558 29087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:03:02.097785 29087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:03:02.102375 29093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:02.102348 29095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:02.102442 29087 server_base.cc:1034] running on GCE node
W20260620 22:03:02.102355 29092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:02.102825 29087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:03:02.103096 29087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:03:02.104230 29087 hybrid_clock.cc:648] HybridClock initialized: now 1781992982104220 us; error 27 us; skew 500 ppm
Jun 20 22:03:02 dist-test-slave-gl2g krb5kdc[29074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992982, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:03:02.106681 29087 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:03:02.107131 29087 webserver.cc:460] Webserver started at http://127.6.241.190:38285/ using document root <none> and password file <none>
I20260620 22:03:02.107347 29087 fs_manager.cc:359] Metadata directory not provided
I20260620 22:03:02.107391 29087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:03:02.107534 29087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:03:02.108546 29087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "ab28b127725e4c7ebf75423525dc1b58"
format_stamp: "Formatted at 2026-06-20 22:03:02 on dist-test-slave-gl2g"
server_key: "27281b61c2c7a152cb88f37c40cfe36a"
server_key_iv: "bac0e250ad5fe1e36b1c4aacbe15d911"
server_key_version: "encryptionkey@0"
I20260620 22:03:02.108899 29087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "ab28b127725e4c7ebf75423525dc1b58"
format_stamp: "Formatted at 2026-06-20 22:03:02 on dist-test-slave-gl2g"
server_key: "27281b61c2c7a152cb88f37c40cfe36a"
server_key_iv: "bac0e250ad5fe1e36b1c4aacbe15d911"
server_key_version: "encryptionkey@0"
I20260620 22:03:02.110236 29087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:03:02.110980 29102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:02.111167 29087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:02.111261 29087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "ab28b127725e4c7ebf75423525dc1b58"
format_stamp: "Formatted at 2026-06-20 22:03:02 on dist-test-slave-gl2g"
server_key: "27281b61c2c7a152cb88f37c40cfe36a"
server_key_iv: "bac0e250ad5fe1e36b1c4aacbe15d911"
server_key_version: "encryptionkey@0"
I20260620 22:03:02.111330 29087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:03:02.123889 29087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:03:02.126022 29087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:03:02.126187 29087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:03:02.131390 29087 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:34687
I20260620 22:03:02.131425 29154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:34687 every 8 connection(s)
I20260620 22:03:02.131767 29087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:03:02.132597 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:03:02.134835 29155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58: Bootstrap starting.
I20260620 22:03:02.135382 29155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58: Neither blocks nor log segments found. Creating new log.
I20260620 22:03:02.135665 29155 log.cc:824] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58: Log is configured to *not* fsync() on all Append() calls
I20260620 22:03:02.136302 29155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58: No bootstrap required, opened a new log
I20260620 22:03:02.137856 29155 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab28b127725e4c7ebf75423525dc1b58" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34687 } }
I20260620 22:03:02.137988 29155 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:03:02.138012 29155 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab28b127725e4c7ebf75423525dc1b58, State: Initialized, Role: FOLLOWER
I20260620 22:03:02.138203 29155 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab28b127725e4c7ebf75423525dc1b58" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34687 } }
I20260620 22:03:02.138288 29155 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:03:02.138330 29155 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:03:02.138377 29155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:03:02.139249 29155 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab28b127725e4c7ebf75423525dc1b58" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34687 } }
I20260620 22:03:02.139334 29155 leader_election.cc:302] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab28b127725e4c7ebf75423525dc1b58; no voters:
I20260620 22:03:02.139497 29155 leader_election.cc:288] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:03:02.139540 29158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:03:02.139802 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 29087
I20260620 22:03:02.139907 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:03:02.140081 7110 external_mini_cluster.cc:1428] Setting key 0d02314be8ed8b78e1a2d9566ae5c940
I20260620 22:03:02.139739 29155 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:03:02.140350 29158 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [term 1 LEADER]: Becoming Leader. State: Replica: ab28b127725e4c7ebf75423525dc1b58, State: Running, Role: LEADER
I20260620 22:03:02.140483 29158 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab28b127725e4c7ebf75423525dc1b58" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34687 } }
I20260620 22:03:02.140919 29160 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab28b127725e4c7ebf75423525dc1b58. Latest consensus state: current_term: 1 leader_uuid: "ab28b127725e4c7ebf75423525dc1b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab28b127725e4c7ebf75423525dc1b58" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34687 } } }
I20260620 22:03:02.141038 29160 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:02.141251 29169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:03:02.141426 29159 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab28b127725e4c7ebf75423525dc1b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab28b127725e4c7ebf75423525dc1b58" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 34687 } } }
I20260620 22:03:02.141578 29159 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58 [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:02.141786 29169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:03:02.143399 29169 catalog_manager.cc:1347] Generated new cluster ID: e61756de019c4c79a127e421278e7f83
I20260620 22:03:02.143453 29169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:03:02.146116 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:02.140730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60061 (local address 127.6.241.190:34687)
0620 22:03:02.141380 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:02.141389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:02.141419 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:03:02.141528 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:02.141535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:02.141602 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:02.141740 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:02.142106 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:02.143408 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:02.144211 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:02.144606 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:02.144723 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:02.145455 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:02.145458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:02.145458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:03:02.145640 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:02.145669 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:03:02.145679 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:02.145749 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:02.145822 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 22:03:02.145884 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":79,"threads_started":1}
I20260620 22:03:02.146960 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:03:02.153388 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:02.148248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39231 (local address 127.6.241.190:34687)
0620 22:03:02.148284 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:02.148286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:02.150843 (+ 2557us) server_negotiation.cc:407] Connection header received
0620 22:03:02.150914 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:02.150915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:02.150925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:02.150969 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:02.151085 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:02.151552 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:02.152254 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:02.152461 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:02.152536 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:02.153156 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:02.153157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:02.153157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:03:02.153163 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:02.153177 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:03:02.153180 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:02.153229 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:02.153274 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 22:03:02.153297 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:03:02.172633 29169 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:03:02.173162 29169 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:03:02.179432 29169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab28b127725e4c7ebf75423525dc1b58: Generated new TSK 0
I20260620 22:03:02.179616 29169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:03:02.187610 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 29087
2026-06-20T22:03:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:03:02 dist-test-slave-gl2g krb5kdc[29186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:03:02 dist-test-slave-gl2g krb5kdc[29186](info): set up 2 sockets
Jun 20 22:03:02 dist-test-slave-gl2g krb5kdc[29186](info): commencing operation
krb5kdc: starting...
W20260620 22:03:03.218839 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:03:03 dist-test-slave-gl2g krb5kdc[29186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:03:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:03:03.332998 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:35025
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:35025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:03:03.413270 29199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:03:03.413509 29199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:03:03.413559 29199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:03:03.414716 29199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:03:03.414784 29199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:03:03.414808 29199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:03:03.414835 29199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:03:03.414867 29199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:03:03.416272 29199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:35025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:35025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:03:03.416531 29199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:03:03.416751 29199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:03:03.421856 29199 server_base.cc:1034] running on GCE node
W20260620 22:03:03.421802 29205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:03.421974 29204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:03.421797 29207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:03.422238 29199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:03:03.422508 29199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:03:03.423645 29199 hybrid_clock.cc:648] HybridClock initialized: now 1781992983423641 us; error 39 us; skew 500 ppm
Jun 20 22:03:03 dist-test-slave-gl2g krb5kdc[29186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992983, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:03:03.431605 29199 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:03:03.432024 29199 webserver.cc:460] Webserver started at http://127.6.241.190:44431/ using document root <none> and password file <none>
I20260620 22:03:03.432246 29199 fs_manager.cc:359] Metadata directory not provided
I20260620 22:03:03.432315 29199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:03:03.432462 29199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:03:03.433483 29199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "6cde6fd5973f42e48238028b0724612f"
format_stamp: "Formatted at 2026-06-20 22:03:03 on dist-test-slave-gl2g"
server_key: "03c58920208cb687fc078e607bbdce3d"
server_key_iv: "8aa81bdc3ca22e54ffc54c81cff8a732"
server_key_version: "encryptionkey@0"
I20260620 22:03:03.433833 29199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "6cde6fd5973f42e48238028b0724612f"
format_stamp: "Formatted at 2026-06-20 22:03:03 on dist-test-slave-gl2g"
server_key: "03c58920208cb687fc078e607bbdce3d"
server_key_iv: "8aa81bdc3ca22e54ffc54c81cff8a732"
server_key_version: "encryptionkey@0"
I20260620 22:03:03.435199 29199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:03:03.435894 29214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:03.436213 29199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:03:03.436311 29199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "6cde6fd5973f42e48238028b0724612f"
format_stamp: "Formatted at 2026-06-20 22:03:03 on dist-test-slave-gl2g"
server_key: "03c58920208cb687fc078e607bbdce3d"
server_key_iv: "8aa81bdc3ca22e54ffc54c81cff8a732"
server_key_version: "encryptionkey@0"
I20260620 22:03:03.436379 29199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:03:03.439651 29199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:03:03.441412 29199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:03:03.441573 29199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:03:03.445523 29199 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:35025
I20260620 22:03:03.445560 29266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:35025 every 8 connection(s)
I20260620 22:03:03.445871 29199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:03:03.446934 29267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:03:03.447854 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 29199
I20260620 22:03:03.447976 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:03:03.448168 7110 external_mini_cluster.cc:1428] Setting key 29efa30a0aa69cadd62da44a5197e417
I20260620 22:03:03.449394 29267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f: Bootstrap starting.
I20260620 22:03:03.450078 29267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f: Neither blocks nor log segments found. Creating new log.
I20260620 22:03:03.450402 29267 log.cc:824] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:03:03.450547 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:03.448665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46841 (local address 127.6.241.190:35025)
0620 22:03:03.448848 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:03.448857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:03.448890 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:03:03.449287 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:03.449293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:03.449316 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:03.449463 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:03.450075 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:03.450078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:03.450078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:03:03.450082 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:03.450120 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:03:03.450137 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:03.450240 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:03.450316 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 22:03:03.450356 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260620 22:03:03.451197 29267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f: No bootstrap required, opened a new log
I20260620 22:03:03.451547 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:03:03.453506 29267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cde6fd5973f42e48238028b0724612f" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35025 } }
I20260620 22:03:03.453626 29267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:03:03.453647 29267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cde6fd5973f42e48238028b0724612f, State: Initialized, Role: FOLLOWER
I20260620 22:03:03.453824 29267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cde6fd5973f42e48238028b0724612f" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35025 } }
I20260620 22:03:03.453935 29267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:03:03.453970 29267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:03:03.454020 29267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:03:03.454774 29267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cde6fd5973f42e48238028b0724612f" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35025 } }
I20260620 22:03:03.454833 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:03.453743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34681 (local address 127.6.241.190:35025)
0620 22:03:03.453778 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:03.453780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:03.453926 (+ 146us) server_negotiation.cc:407] Connection header received
0620 22:03:03.453981 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:03.453982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:03.453986 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:03.454043 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:03.454564 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:03.454565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:03.454565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:03:03.454569 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:03.454580 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:03:03.454583 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:03.454690 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:03.454744 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 22:03:03.454754 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:03:03.454914 29267 leader_election.cc:302] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cde6fd5973f42e48238028b0724612f; no voters:
I20260620 22:03:03.455083 29267 leader_election.cc:288] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:03:03.455130 29277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:03:03.455277 29277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [term 1 LEADER]: Becoming Leader. State: Replica: 6cde6fd5973f42e48238028b0724612f, State: Running, Role: LEADER
I20260620 22:03:03.455307 29267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:03:03.455431 29277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cde6fd5973f42e48238028b0724612f" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35025 } }
I20260620 22:03:03.455832 29278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cde6fd5973f42e48238028b0724612f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cde6fd5973f42e48238028b0724612f" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35025 } } }
I20260620 22:03:03.455935 29278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:03.456115 29279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cde6fd5973f42e48238028b0724612f. Latest consensus state: current_term: 1 leader_uuid: "6cde6fd5973f42e48238028b0724612f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cde6fd5973f42e48238028b0724612f" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 35025 } } }
I20260620 22:03:03.456138 29285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:03:03.456169 29279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:03.456599 29285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:03:03.458017 29285 catalog_manager.cc:1347] Generated new cluster ID: 911a345bd0af46a5a46222c73e058ef5
I20260620 22:03:03.458065 29285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:03:03.479526 29285 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:03:03.479703 29285 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:03:03.493101 29285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cde6fd5973f42e48238028b0724612f: Generated new TSK 0
I20260620 22:03:03.493304 29285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:03:03.504760 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 29199
2026-06-20T22:03:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1317 ms)
[----------] 18 tests from AuthTokenIssuingTest (23780 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:03:03 dist-test-slave-gl2g krb5kdc[29298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:03:03 dist-test-slave-gl2g krb5kdc[29298](info): set up 2 sockets
Jun 20 22:03:03 dist-test-slave-gl2g krb5kdc[29298](info): commencing operation
krb5kdc: starting...
W20260620 22:03:04.538751 7110 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:03:04 dist-test-slave-gl2g krb5kdc[29298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:03:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:03:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:03:04.647251 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:38719
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:38719,127.6.241.189:38841,127.6.241.188:41455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:03:04.725596 29311 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:03:04.725826 29311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:03:04.725855 29311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:03:04.725909 29311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:03:04.727123 29311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:03:04.727185 29311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:03:04.727219 29311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:03:04.727259 29311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:03:04.727293 29311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:03:04.728703 29311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:38719,127.6.241.189:38841,127.6.241.188:41455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:38719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:03:04.728965 29311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:03:04.729182 29311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:03:04.733796 29319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:04.733790 29316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:04.733968 29311 server_base.cc:1034] running on GCE node
W20260620 22:03:04.733810 29317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:04.734247 29311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:03:04.734519 29311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:03:04.735661 29311 hybrid_clock.cc:648] HybridClock initialized: now 1781992984735649 us; error 30 us; skew 500 ppm
Jun 20 22:03:04 dist-test-slave-gl2g krb5kdc[29298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:03:04.738068 29311 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:03:04.738428 29311 webserver.cc:460] Webserver started at http://127.6.241.190:33319/ using document root <none> and password file <none>
I20260620 22:03:04.738658 29311 fs_manager.cc:359] Metadata directory not provided
I20260620 22:03:04.738708 29311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:03:04.738799 29311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:03:04.739816 29311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "91af49dec44643cdb8cee2d3d748d691"
format_stamp: "Formatted at 2026-06-20 22:03:04 on dist-test-slave-gl2g"
server_key: "5618ac4936b3a659574fd789e3e1eac0"
server_key_iv: "451661d019c7ce3e41a929874db22861"
server_key_version: "encryptionkey@0"
I20260620 22:03:04.740097 29311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "91af49dec44643cdb8cee2d3d748d691"
format_stamp: "Formatted at 2026-06-20 22:03:04 on dist-test-slave-gl2g"
server_key: "5618ac4936b3a659574fd789e3e1eac0"
server_key_iv: "451661d019c7ce3e41a929874db22861"
server_key_version: "encryptionkey@0"
I20260620 22:03:04.741380 29311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:03:04.742023 29326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:04.742190 29311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:03:04.742304 29311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "91af49dec44643cdb8cee2d3d748d691"
format_stamp: "Formatted at 2026-06-20 22:03:04 on dist-test-slave-gl2g"
server_key: "5618ac4936b3a659574fd789e3e1eac0"
server_key_iv: "451661d019c7ce3e41a929874db22861"
server_key_version: "encryptionkey@0"
I20260620 22:03:04.742378 29311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:03:04.768736 29311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:03:04.771111 29311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:03:04.771270 29311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:03:04.775164 29311 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:38719
I20260620 22:03:04.775202 29378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:38719 every 8 connection(s)
I20260620 22:03:04.775579 29311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:03:04.776538 29379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:03:04.778048 29379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:04.781136 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 29311
I20260620 22:03:04.781266 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:03:04.781458 7110 external_mini_cluster.cc:1428] Setting key 7c3286631c998c737d65fda3c9cbc0ea
Jun 20 22:03:04 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:03:04.787851 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.778780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.6.241.190:38719)
0620 22:03:04.778967 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:04.778971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:04.779003 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:03:04.779479 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:04.779480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:04.779495 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:04.779569 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:04.780065 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.780875 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.781877 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.782378 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.782493 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:04.784744 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:04.784750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:04.784751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:04.784757 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:04.786265 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:04.786699 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:04.786702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:04.786702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:04.786783 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:04.787015 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:04.787016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:04.787016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:04.787214 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:04.787304 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:04.787493 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 22:03:04.787682 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":162,"thread_start_us":115,"threads_started":1}
I20260620 22:03:04.787855 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.778716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:38719 (local address 127.0.0.1:59410)
0620 22:03:04.778886 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.778930 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:04.779360 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:04.779595 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:04.779598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:04.779841 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:04.779989 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:04.779993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.780952 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.780953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:04.781786 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:04.781789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.782485 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.782485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:04.782537 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:04.783096 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:04.783107 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:04.784643 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:04.786370 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:04.786376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:04.786378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:04.786569 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:04.786854 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:04.786855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:04.786855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:04.786944 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:04.787286 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:04.787291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:04.787349 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 22:03:04.787448 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 22:03:04.787624 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260620 22:03:04.788518 29379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:04.788866 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.788740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:38841 (local address 127.0.0.1:54168)
0620 22:03:04.788787 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.788844 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:03:04.788995 29329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.6.241.189@KRBTEST.COM; defaulting to no policy
I20260620 22:03:04.789459 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.789354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:38841 (local address 127.0.0.1:54184)
0620 22:03:04.789408 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.789436 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 22:03:04.789542 29379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.189:38841: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab.
Entry for principal kudu/127.6.241.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab.
I20260620 22:03:04.803599 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.189:38841
--webserver_interface=127.6.241.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab
--principal=kudu/127.6.241.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:38719,127.6.241.189:38841,127.6.241.188:41455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:03:04.838757 29379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } attempt: 1
I20260620 22:03:04.839226 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.839094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:38841 (local address 127.0.0.1:54188)
0620 22:03:04.839156 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.839196 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 22:03:04.839665 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.839564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:38841 (local address 127.0.0.1:54198)
0620 22:03:04.839621 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.839647 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 22:03:04.839779 29379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.189:38841: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
W20260620 22:03:04.882400 29386 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:03:04.882616 29386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:03:04.882634 29386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:03:04.882654 29386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:03:04.883811 29386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:03:04.883857 29386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:03:04.883872 29386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:03:04.883885 29386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:03:04.883944 29386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:03:04.885511 29386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:38719,127.6.241.189:38841,127.6.241.188:41455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab
--principal=kudu/127.6.241.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.189:38841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.241.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:03:04.885766 29386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:03:04.885986 29386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:03:04.890811 29391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:04.890897 29386 server_base.cc:1034] running on GCE node
W20260620 22:03:04.890818 29394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:04.890818 29392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:04.891266 29386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:03:04.891566 29386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:03:04.892738 29386 hybrid_clock.cc:648] HybridClock initialized: now 1781992984892722 us; error 45 us; skew 500 ppm
Jun 20 22:03:04 dist-test-slave-gl2g krb5kdc[29298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:03:04.895284 29386 init.cc:377] Logged in from keytab as kudu/127.6.241.189@KRBTEST.COM (short username kudu)
I20260620 22:03:04.895720 29386 webserver.cc:460] Webserver started at http://127.6.241.189:44521/ using document root <none> and password file <none>
I20260620 22:03:04.895946 29386 fs_manager.cc:359] Metadata directory not provided
I20260620 22:03:04.896015 29386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:03:04.896163 29386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:03:04.897212 29386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/data/instance:
uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec"
format_stamp: "Formatted at 2026-06-20 22:03:04 on dist-test-slave-gl2g"
server_key: "05fa629224d09726405ce36fdaf7f126"
server_key_iv: "cf1dc304f89fb992609e8e441079ed05"
server_key_version: "encryptionkey@0"
I20260620 22:03:04.897590 29386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/wal/instance:
uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec"
format_stamp: "Formatted at 2026-06-20 22:03:04 on dist-test-slave-gl2g"
server_key: "05fa629224d09726405ce36fdaf7f126"
server_key_iv: "cf1dc304f89fb992609e8e441079ed05"
server_key_version: "encryptionkey@0"
I20260620 22:03:04.898906 29386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:03:04.899580 29401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:04.899780 29386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:03:04.899868 29386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/wal
uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec"
format_stamp: "Formatted at 2026-06-20 22:03:04 on dist-test-slave-gl2g"
server_key: "05fa629224d09726405ce36fdaf7f126"
server_key_iv: "cf1dc304f89fb992609e8e441079ed05"
server_key_version: "encryptionkey@0"
I20260620 22:03:04.899940 29386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:03:04.907915 29379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } attempt: 2
I20260620 22:03:04.908391 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.908250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:38841 (local address 127.0.0.1:54204)
0620 22:03:04.908320 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.908361 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 22:03:04.908715 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.908622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:38841 (local address 127.0.0.1:54216)
0620 22:03:04.908666 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.908690 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:03:04.908818 29379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.189:38841: Network error: Client connection negotiation failed: client connection to 127.6.241.189:38841: connect: Connection refused (error 111)
I20260620 22:03:04.922268 29386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:03:04.924527 29386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:03:04.924642 29386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:03:04.928656 29386 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.189:38841
I20260620 22:03:04.928709 29453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.189:38841 every 8 connection(s)
I20260620 22:03:04.929067 29386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
I20260620 22:03:04.930073 29454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:03:04.931587 29454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:03:04 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:03:04.938040 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 29386
I20260620 22:03:04.938438 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-1/wal/instance
I20260620 22:03:04.938694 7110 external_mini_cluster.cc:1428] Setting key 2fd048b80efabd0c6a76c945f0dddb0c
I20260620 22:03:04.940472 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.932263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.6.241.190:38719)
0620 22:03:04.932323 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:04.932325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:04.932528 (+ 203us) server_negotiation.cc:407] Connection header received
0620 22:03:04.932989 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:04.932990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:04.932999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:04.933064 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:04.933639 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.934117 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.935093 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.935294 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.935348 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:04.937379 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:04.937381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:04.937381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:04.937387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:04.939120 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:04.939485 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:04.939487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:04.939487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:04.939551 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:04.939783 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:04.939784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:04.939785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:04.939933 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:04.940001 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:04.940342 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 22:03:04.940408 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:03:04.940764 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.932235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:38719 (local address 127.0.0.1:59412)
0620 22:03:04.932403 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.932444 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:04.932902 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:04.933117 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:04.933120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:04.933365 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:04.933555 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:04.933560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.934197 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.934198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:04.935024 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:04.935026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.935385 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.935386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:04.935493 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:04.935939 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:04.935946 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:04.937289 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:04.939195 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:04.939201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:04.939202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:04.939412 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:04.939620 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:04.939621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:04.939621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:04.939716 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:04.940016 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:04.940023 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:04.940199 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 22:03:04.940319 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 22:03:04.940485 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260620 22:03:04.941396 29454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:03:04 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.241.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.188@KRBTEST.COM" created.
I20260620 22:03:04.950543 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.941728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.6.241.189:38841)
0620 22:03:04.941907 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:04.941911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:04.941927 (+ 16us) server_negotiation.cc:407] Connection header received
0620 22:03:04.941943 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:04.941944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:04.941954 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:04.942136 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:04.942424 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.943631 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.944296 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.944845 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.944919 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:04.946943 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:04.946950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:04.946951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:04.946958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:04.949275 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:04.949682 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:04.949685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:04.949685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:04.949744 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:04.949914 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:04.949915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:04.949915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:04.950085 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:04.950168 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:04.950390 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 22:03:04.950458 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260620 22:03:04.950556 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.941649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:38841 (local address 127.0.0.1:54230)
0620 22:03:04.941697 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.941707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:04.941750 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:04.942138 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:04.942139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:04.942268 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:04.942314 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:04.942315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.943726 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.943727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:04.944216 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:04.944218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:04.944939 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:04.944939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:04.945011 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:04.945396 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:04.945400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:04.946849 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:04.949364 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:04.949364 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:04.949365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:04.949589 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:04.949807 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:04.949807 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:04.949807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:04.949860 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:04.950190 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:04.950190 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:04.950226 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 22:03:04.950286 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:03:04.950382 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:03:04.951510 29454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:04.951867 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.951737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44318)
0620 22:03:04.951789 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.951836 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 22:03:04.952001 29405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
I20260620 22:03:04.952471 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:04.952352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44320)
0620 22:03:04.952409 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:04.952440 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 22:03:04.952587 29454 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:41455: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab.
Entry for principal kudu/127.6.241.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab.
I20260620 22:03:04.962898 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.188:41455
--webserver_interface=127.6.241.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab
--principal=kudu/127.6.241.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:38719,127.6.241.189:38841,127.6.241.188:41455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:03:04.999985 29379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } attempt: 3
I20260620 22:03:05.001814 29454 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } attempt: 1
I20260620 22:03:05.002194 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.002057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44330)
0620 22:03:05.002117 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.002156 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 22:03:05.002698 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.002580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44338)
0620 22:03:05.002642 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.002668 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 22:03:05.002929 29454 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:41455: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
I20260620 22:03:05.007649 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.000363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:38841 (local address 127.0.0.1:54242)
0620 22:03:05.000429 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.000444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:05.000518 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:05.000673 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:05.000674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:05.000834 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:05.000896 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.000898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.001609 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.001610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.002308 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.002311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.002885 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.002886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.002974 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.003444 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:05.003450 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:05.004884 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:05.006766 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.006767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.006767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.006921 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.007114 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.007115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.007115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.007166 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.007416 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:05.007417 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:05.007453 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 22:03:05.007502 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:03:05.007571 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 22:03:05.007681 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.000442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.6.241.189:38841)
0620 22:03:05.000505 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.000508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.000520 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:03:05.000584 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.000586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.000598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.000668 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.000981 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.001518 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.002396 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.002787 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.002875 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.005001 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.005003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.005003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.005010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.006683 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.006988 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.006989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.006990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.007049 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.007226 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.007226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.007227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.007355 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.007423 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.007524 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.007600 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:03:05.008351 29379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:05.008683 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.008572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44344)
0620 22:03:05.008637 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.008661 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 22:03:05.008992 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.008938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44352)
0620 22:03:05.008961 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.008980 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 22:03:05.009090 29379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:41455: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
I20260620 22:03:05.040241 29379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } attempt: 1
I20260620 22:03:05.040658 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.040529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44360)
0620 22:03:05.040594 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.040631 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 22:03:05.041033 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.040942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44376)
0620 22:03:05.040991 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.041016 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 22:03:05.041134 29379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:41455: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
W20260620 22:03:05.044315 29461 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:03:05.044548 29461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:03:05.044586 29461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:03:05.044620 29461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:03:05.045750 29461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:03:05.045816 29461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:03:05.045842 29461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:03:05.045871 29461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:03:05.045898 29461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:03:05.047434 29461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:38719,127.6.241.189:38841,127.6.241.188:41455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab
--principal=kudu/127.6.241.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.188:41455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.241.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:03:05.047703 29461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:03:05.047927 29461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:03:05.052616 29469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:05.052644 29467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:05.052634 29466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:05.052893 29461 server_base.cc:1034] running on GCE node
I20260620 22:03:05.053131 29461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:03:05.053417 29461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:03:05.054580 29461 hybrid_clock.cc:648] HybridClock initialized: now 1781992985054567 us; error 24 us; skew 500 ppm
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:03:05.056928 29461 init.cc:377] Logged in from keytab as kudu/127.6.241.188@KRBTEST.COM (short username kudu)
I20260620 22:03:05.057392 29461 webserver.cc:460] Webserver started at http://127.6.241.188:42857/ using document root <none> and password file <none>
I20260620 22:03:05.057605 29461 fs_manager.cc:359] Metadata directory not provided
I20260620 22:03:05.057646 29461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:03:05.057732 29461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:03:05.058794 29461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/data/instance:
uuid: "6a1fac32e07a4ad08a1a66027d2c9881"
format_stamp: "Formatted at 2026-06-20 22:03:05 on dist-test-slave-gl2g"
server_key: "2d1658a8b9c6db1b31c77e1f5af97419"
server_key_iv: "4bfba4083a2fa81a2ea140961d61211c"
server_key_version: "encryptionkey@0"
I20260620 22:03:05.059157 29461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/wal/instance:
uuid: "6a1fac32e07a4ad08a1a66027d2c9881"
format_stamp: "Formatted at 2026-06-20 22:03:05 on dist-test-slave-gl2g"
server_key: "2d1658a8b9c6db1b31c77e1f5af97419"
server_key_iv: "4bfba4083a2fa81a2ea140961d61211c"
server_key_version: "encryptionkey@0"
I20260620 22:03:05.060391 29461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:03:05.061086 29476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:05.061297 29461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:03:05.061381 29461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/wal
uuid: "6a1fac32e07a4ad08a1a66027d2c9881"
format_stamp: "Formatted at 2026-06-20 22:03:05 on dist-test-slave-gl2g"
server_key: "2d1658a8b9c6db1b31c77e1f5af97419"
server_key_iv: "4bfba4083a2fa81a2ea140961d61211c"
server_key_version: "encryptionkey@0"
I20260620 22:03:05.061437 29461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:03:05.069418 29461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:03:05.071081 29454 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } attempt: 2
I20260620 22:03:05.071292 29461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:03:05.071445 29461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:03:05.071489 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.071366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44382)
0620 22:03:05.071428 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.071462 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 22:03:05.071841 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.071692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44398)
0620 22:03:05.071772 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.071809 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 22:03:05.071950 29454 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:41455: Network error: Client connection negotiation failed: client connection to 127.6.241.188:41455: connect: Connection refused (error 111)
I20260620 22:03:05.075323 29461 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.188:41455
I20260620 22:03:05.075356 29528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.188:41455 every 8 connection(s)
I20260620 22:03:05.075692 29461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
I20260620 22:03:05.076673 29529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:03:05.076917 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 29461
I20260620 22:03:05.077051 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992718478901-7110-0/minicluster-data/master-2/wal/instance
I20260620 22:03:05.077266 7110 external_mini_cluster.cc:1428] Setting key 073c728293ecf1311bed543570d35e33
I20260620 22:03:05.078545 29529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:03:05.087822 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.079282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.6.241.190:38719)
0620 22:03:05.079351 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.079353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.079470 (+ 117us) server_negotiation.cc:407] Connection header received
0620 22:03:05.079524 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.079525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.079536 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.079602 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.080160 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.080619 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.081590 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.081902 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.081966 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.084246 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.084247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.084247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.084253 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.086264 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.086821 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.086822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.086823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.086896 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.087136 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.087137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.087137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.087293 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.087365 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.087681 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.087758 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:03:05.088022 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.079222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:38719 (local address 127.0.0.1:59414)
0620 22:03:05.079370 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.079407 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:05.079470 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:05.079643 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:05.079646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:05.079893 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:05.080043 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.080047 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.080696 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.080697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.081515 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.081517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.081996 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.081997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.082121 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.082720 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:05.082727 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:05.084165 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:05.086346 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.086352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.086353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.086733 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.086963 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.086964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.086964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.087058 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.087361 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:05.087366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:05.087534 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 22:03:05.087627 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 22:03:05.087799 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260620 22:03:05.088549 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.077781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.6.241.188:41455)
0620 22:03:05.078746 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.078756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.078791 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:03:05.079289 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.079298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.079366 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.079507 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.080112 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.081063 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.081877 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.082297 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.082369 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.084734 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.084742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.084742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.084750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.086751 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.087406 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.087408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.087408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.087464 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.087676 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.087677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.087677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.087826 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.087956 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.088366 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.088448 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":81,"threads_started":1}
I20260620 22:03:05.088598 29529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:03:05.095147 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.088837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:38841 (local address 127.0.0.1:54252)
0620 22:03:05.088877 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.088887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:05.088918 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:05.089058 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:05.089058 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:05.089207 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:05.089252 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.089254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.089956 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.089957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.090429 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.090431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.090867 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.090868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.090934 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.091336 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:05.091341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:05.092492 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:05.094072 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.094072 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.094073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.094308 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.094516 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.094517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.094517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.094569 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.094916 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:05.094916 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:05.094959 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:03:05.095014 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:03:05.095082 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:03:05.095340 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.088909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.6.241.189:38841)
0620 22:03:05.088964 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.088966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.088975 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:03:05.088998 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.088999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.089008 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.089056 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.089313 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.089877 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.090508 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.090796 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.090870 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.092574 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.092575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.092576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.092583 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.093988 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.094385 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.094386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.094387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.094443 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.094696 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.094697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.094698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.094843 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.094912 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.095062 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.095137 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:03:05.095661 29529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:05.096788 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.089400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.6.241.190:38719)
0620 22:03:05.089437 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.089439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.089446 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:03:05.089500 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.089502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.089510 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.089561 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.090020 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.090449 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.091243 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.091535 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.091609 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.093904 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.093905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.093906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.093913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.095635 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.096020 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.096021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.096022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.096082 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.096301 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.096302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.096303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.096433 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.096591 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.096669 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.096729 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
I20260620 22:03:05.102836 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.095947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44426)
0620 22:03:05.096045 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.096056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:05.096098 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:05.096231 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:05.096232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:05.096383 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:05.096422 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.096423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.097302 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.097303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.097931 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.097934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.098295 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.098295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.098342 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.098768 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:05.098773 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:05.100101 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:05.101677 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.101678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.101678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.101886 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.102109 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.102111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.102112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.102163 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.102407 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:05.102408 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:05.102443 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 22:03:05.102532 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 22:03:05.102682 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 22:03:05.102885 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.096001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.6.241.188:41455)
0620 22:03:05.096035 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.096036 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.096091 (+ 55us) server_negotiation.cc:407] Connection header received
0620 22:03:05.096157 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.096158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.096168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.096229 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.096522 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.097245 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.098008 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.098220 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.098298 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.100184 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.100202 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.100202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.100210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.101596 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.101957 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.101958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.101958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.102004 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.102228 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.102229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.102230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.102342 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.102414 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.102553 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.102733 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":17}
I20260620 22:03:05.105037 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881: Bootstrap starting.
I20260620 22:03:05.105213 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.097492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.6.241.189:38841)
0620 22:03:05.097545 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.097547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.097554 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:03:05.097590 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.097591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.097599 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.097647 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.098069 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.098564 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.099494 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.099799 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.099874 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.102410 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.102411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.102411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.102418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.104174 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.104474 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.104475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.104476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.104535 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.104731 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.104732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.104732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.104867 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.105024 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.105064 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.105132 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:03:05.105984 29529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881: Neither blocks nor log segments found. Creating new log.
I20260620 22:03:05.106353 29529 log.cc:824] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881: Log is configured to *not* fsync() on all Append() calls
I20260620 22:03:05.106637 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:03:05.107147 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881: No bootstrap required, opened a new log
I20260620 22:03:05.107530 29353 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "91af49dec44643cdb8cee2d3d748d691"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59426
I20260620 22:03:05.108690 29529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } }
I20260620 22:03:05.108817 29529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1fac32e07a4ad08a1a66027d2c9881, State: Initialized, Role: FOLLOWER
I20260620 22:03:05.108906 29529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } }
I20260620 22:03:05.109246 29529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:03:05.109215 29535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.109325 29535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:03:05.111011 29546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:03:05.111068 29546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:03:05.111248 29461 master_runner.cc:186] Error getting master registration for 127.6.241.190:38719: OK, instance_id { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" instance_seqno: 1781992984774002 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:03:05.113286 29461 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:03:05.116263 29379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } attempt: 2
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:03:05.124185 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.116538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44430)
0620 22:03:05.116575 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.116586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:05.116641 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:05.116765 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:05.116766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:05.116931 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:05.116979 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.116981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.117886 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.117886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.118732 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.118734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.119207 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.119208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.119260 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.119745 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:05.119750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:05.120986 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:05.123049 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.123050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.123050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.123294 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.123505 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.123506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.123507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.123568 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.123866 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:05.123867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:05.123913 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 22:03:05.123980 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 22:03:05.124068 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 22:03:05.124204 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.116593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.6.241.188:41455)
0620 22:03:05.116630 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.116631 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.116640 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:03:05.116702 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.116703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.116712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.116764 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.117053 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.117798 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.118828 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.119127 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.119202 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.121083 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.121085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.121085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.121092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.122843 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.123373 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.123374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.123375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.123434 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.123632 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.123633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.123633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.123798 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.123872 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.124020 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.124095 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:03:05.125710 29379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691: Bootstrap starting.
I20260620 22:03:05.126485 29379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691: Neither blocks nor log segments found. Creating new log.
I20260620 22:03:05.126819 29379 log.cc:824] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691: Log is configured to *not* fsync() on all Append() calls
I20260620 22:03:05.127658 29379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691: No bootstrap required, opened a new log
I20260620 22:03:05.129004 29379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } }
I20260620 22:03:05.129142 29379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91af49dec44643cdb8cee2d3d748d691, State: Initialized, Role: FOLLOWER
I20260620 22:03:05.129227 29379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } }
I20260620 22:03:05.129542 29379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:03:05.129499 29549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.129568 29353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 22:03:05.129590 29549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:05.129623 29353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:03:05.130353 29353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } }
I20260620 22:03:05.131057 29353 leader_election.cc:288] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [CANDIDATE]: Term 1 election: Requested vote from peers bbd7f1371b0d4dde8f7b6ee2d3c39aec (127.6.241.189:38841), 6a1fac32e07a4ad08a1a66027d2c9881 (127.6.241.188:41455)
I20260620 22:03:05.131377 29503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91af49dec44643cdb8cee2d3d748d691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6a1fac32e07a4ad08a1a66027d2c9881"
I20260620 22:03:05.131520 29503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [term 0 FOLLOWER]: Advancing to term 1
W20260620 22:03:05.131805 29560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:03:05.131876 29560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:03:05.132426 29503 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91af49dec44643cdb8cee2d3d748d691 in term 1.
I20260620 22:03:05.132725 29330 leader_election.cc:302] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1fac32e07a4ad08a1a66027d2c9881, 91af49dec44643cdb8cee2d3d748d691; no voters:
I20260620 22:03:05.132874 29549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:03:05.133010 29549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [term 1 LEADER]: Becoming Leader. State: Replica: 91af49dec44643cdb8cee2d3d748d691, State: Running, Role: LEADER
I20260620 22:03:05.133096 29549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } }
I20260620 22:03:05.133518 29549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91af49dec44643cdb8cee2d3d748d691. Latest consensus state: current_term: 1 leader_uuid: "91af49dec44643cdb8cee2d3d748d691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.133603 29549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:05.134536 29311 master_runner.cc:186] Error getting master registration for 127.6.241.189:38841: OK, instance_id { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" instance_seqno: 1781992984927462 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:03:05.136775 29311 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:03:05.139127 29568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:03:05.139329 29568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:03:05.141161 29503 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [term 1 FOLLOWER]: Refusing update from remote peer 91af49dec44643cdb8cee2d3d748d691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:03:05.141553 29549 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:03:05.142871 29535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91af49dec44643cdb8cee2d3d748d691. Latest consensus state: current_term: 1 leader_uuid: "91af49dec44643cdb8cee2d3d748d691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.142971 29535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:05.146487 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.133711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.6.241.188:41455)
0620 22:03:05.133761 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.133763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.134966 (+ 1203us) server_negotiation.cc:407] Connection header received
0620 22:03:05.135124 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.135125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.135163 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.135195 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.136473 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.137581 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.139217 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.139669 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.139734 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.142050 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.142052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.142053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.142060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.144528 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.145139 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.145140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.145141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.145220 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.145546 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.145547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.145547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.145755 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.145876 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.146208 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.146307 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:03:05.152987 29549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91af49dec44643cdb8cee2d3d748d691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.153083 29549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:05.156371 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.134533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.6.241.190:38719)
0620 22:03:05.134579 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.134581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.147205 (+ 12624us) server_negotiation.cc:407] Connection header received
0620 22:03:05.147431 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.147432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.147455 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.147486 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.148513 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.149244 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.150692 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.150985 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.151023 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.153173 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.153186 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.153186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.153193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.154809 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.155339 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.155340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.155341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.155404 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.155691 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.155692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.155692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.155825 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.155896 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.156205 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.156282 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:03:05.156632 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.134378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.6.241.189:38841)
0620 22:03:05.134417 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.134419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.140329 (+ 5910us) server_negotiation.cc:407] Connection header received
0620 22:03:05.140429 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.140430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.140441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.140470 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.141818 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.142450 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.151402 (+ 8952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.151682 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.151752 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.153857 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.153858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.153858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.153866 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.155471 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.155832 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.155833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.155833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.155890 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.156076 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.156077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.156078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.156202 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.156283 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.156444 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.156518 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:03:05.160358 29535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91af49dec44643cdb8cee2d3d748d691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.160452 29535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:05.163090 29454 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } attempt: 3
I20260620 22:03:05.166630 29572 mvcc.cc:205] Tried to move back new op lower bound from 7299043267135164416 to 7299043267106021376. Current Snapshot: MvccSnapshot[applied={T|T < 7299043267135164416}]
I20260620 22:03:05.167286 29568 catalog_manager.cc:1347] Generated new cluster ID: 661a98ade6c247a991f682d45ca99532
I20260620 22:03:05.167339 29568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:03:05 dist-test-slave-gl2g krb5kdc[29298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:03:05.171401 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.163375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:41455 (local address 127.0.0.1:44448)
0620 22:03:05.163450 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:05.163459 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:05.163493 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:05.163743 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:05.163744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:05.163898 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:05.163941 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.163943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.164931 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.164932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.165721 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:05.165723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.166250 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.166251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:05.166320 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.166738 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:05.166743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:05.168494 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:05.170263 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.170264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.170264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.170476 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.170729 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:05.170730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:05.170730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:05.170799 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:05.171096 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:05.171097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:05.171151 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 22:03:05.171213 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:03:05.171301 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 22:03:05.171764 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:05.163561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.6.241.188:41455)
0620 22:03:05.163600 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.163628 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.163634 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:03:05.163653 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.163654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.163661 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.163705 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.164018 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.164813 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.165842 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:05.166133 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:05.166190 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:05.168598 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:05.168599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:05.168599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:05.168606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:05.170159 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.170560 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.170561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.170562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.170646 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:05.170874 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:05.170875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:05.170876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:05.171007 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:05.171060 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:05.171583 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 22:03:05.171676 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:03:05.173259 29454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec: Bootstrap starting.
I20260620 22:03:05.173976 29454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec: Neither blocks nor log segments found. Creating new log.
I20260620 22:03:05.174273 29454 log.cc:824] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec: Log is configured to *not* fsync() on all Append() calls
I20260620 22:03:05.175525 29454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec: No bootstrap required, opened a new log
I20260620 22:03:05.177469 29454 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } }
I20260620 22:03:05.177642 29454 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd7f1371b0d4dde8f7b6ee2d3c39aec, State: Initialized, Role: FOLLOWER
I20260620 22:03:05.177728 29454 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } }
I20260620 22:03:05.178053 29454 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:03:05.178125 29427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:03:05.178956 29427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [term 1 FOLLOWER]: Refusing update from remote peer 91af49dec44643cdb8cee2d3d748d691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:03:05.179306 29562 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:03:05.179483 29428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91af49dec44643cdb8cee2d3d748d691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec"
I20260620 22:03:05.180253 29428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91af49dec44643cdb8cee2d3d748d691 in term 1.
I20260620 22:03:05.183892 29579 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91af49dec44643cdb8cee2d3d748d691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.183990 29579 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:05.184047 29579 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91af49dec44643cdb8cee2d3d748d691. Latest consensus state: current_term: 1 leader_uuid: "91af49dec44643cdb8cee2d3d748d691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.184124 29579 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:05.184139 29593 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91af49dec44643cdb8cee2d3d748d691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.184211 29593 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:03:05.184667 29594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:03:05.184719 29594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:03:05.185449 29591 mvcc.cc:205] Tried to move back new op lower bound from 7299043267135164416 to 7299043267106021376. Current Snapshot: MvccSnapshot[applied={T|T < 7299043267135164416}]
I20260620 22:03:05.201020 29562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91af49dec44643cdb8cee2d3d748d691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91af49dec44643cdb8cee2d3d748d691" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 38719 } } peers { permanent_uuid: "bbd7f1371b0d4dde8f7b6ee2d3c39aec" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 38841 } } peers { permanent_uuid: "6a1fac32e07a4ad08a1a66027d2c9881" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 41455 } } }
I20260620 22:03:05.201139 29562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691 [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:05.203611 29568 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:03:05.204358 29568 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:03:05.213128 29568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91af49dec44643cdb8cee2d3d748d691: Generated new TSK 0
I20260620 22:03:05.213311 29568 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:03:05.243161 29532 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.241720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.6.241.188:41455)
0620 22:03:05.241780 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.241783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.242444 (+ 661us) server_negotiation.cc:407] Connection header received
0620 22:03:05.242555 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.242556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.242568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.242629 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.243092 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44456: BlockingRecv error: recv got EOF from 127.0.0.1:44456 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 22:03:05.243949 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.242139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.6.241.189:38841)
0620 22:03:05.242179 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.242181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.243409 (+ 1228us) server_negotiation.cc:407] Connection header received
0620 22:03:05.243505 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.243507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.243518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.243548 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.243885 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54292: BlockingRecv error: recv got EOF from 127.0.0.1:54292 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 22:03:05.244869 29382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.244338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.6.241.190:38719)
0620 22:03:05.244381 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.244383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.244393 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:03:05.244446 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.244447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.244459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.244490 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.244807 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59446: BlockingRecv error: recv got EOF from 127.0.0.1:59446 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 22:03:05.247306 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.246538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.6.241.189:38841)
0620 22:03:05.246573 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.246574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.246797 (+ 223us) server_negotiation.cc:407] Connection header received
0620 22:03:05.246902 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.246903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.246920 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.246950 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.247258 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54298: BlockingRecv error: recv got EOF from 127.0.0.1:54298 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 22:03:05.250296 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.249516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.6.241.189:38841)
0620 22:03:05.249549 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.249550 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.249803 (+ 253us) server_negotiation.cc:407] Connection header received
0620 22:03:05.249898 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.249899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.249908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.249936 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.250242 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54302: BlockingRecv error: recv got EOF from 127.0.0.1:54302 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 22:03:05.254151 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.253385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.6.241.189:38841)
0620 22:03:05.253417 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.253418 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.253657 (+ 239us) server_negotiation.cc:407] Connection header received
0620 22:03:05.253753 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.253754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.253763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.253791 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.254107 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54306: BlockingRecv error: recv got EOF from 127.0.0.1:54306 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:03:05.259980 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.259219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.6.241.189:38841)
0620 22:03:05.259251 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.259253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.259490 (+ 237us) server_negotiation.cc:407] Connection header received
0620 22:03:05.259585 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.259586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.259595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.259623 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.259931 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54308: BlockingRecv error: recv got EOF from 127.0.0.1:54308 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 22:03:05.269858 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.269051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.6.241.189:38841)
0620 22:03:05.269090 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.269092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.269344 (+ 252us) server_negotiation.cc:407] Connection header received
0620 22:03:05.269440 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.269441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.269450 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.269480 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.269808 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54324: BlockingRecv error: recv got EOF from 127.0.0.1:54324 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:03:05.287902 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.287089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.6.241.189:38841)
0620 22:03:05.287125 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.287127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.287391 (+ 264us) server_negotiation.cc:407] Connection header received
0620 22:03:05.287484 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.287485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.287495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.287523 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.287847 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54334: BlockingRecv error: recv got EOF from 127.0.0.1:54334 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 22:03:05.321923 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.321161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.6.241.189:38841)
0620 22:03:05.321208 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.321211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.321357 (+ 146us) server_negotiation.cc:407] Connection header received
0620 22:03:05.321432 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.321433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.321445 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.321518 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.321848 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54350: BlockingRecv error: recv got EOF from 127.0.0.1:54350 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 22:03:05.387866 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.387057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.6.241.189:38841)
0620 22:03:05.387141 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.387144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.387247 (+ 103us) server_negotiation.cc:407] Connection header received
0620 22:03:05.387367 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.387368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.387380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.387451 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.387797 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54352: BlockingRecv error: recv got EOF from 127.0.0.1:54352 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 22:03:05.517745 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.516925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.6.241.189:38841)
0620 22:03:05.517005 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.517007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.517109 (+ 102us) server_negotiation.cc:407] Connection header received
0620 22:03:05.517213 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.517214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.517227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.517310 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.517676 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54358: BlockingRecv error: recv got EOF from 127.0.0.1:54358 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 22:03:05.775738 29458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:05.774972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.6.241.189:38841)
0620 22:03:05.775060 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:05.775063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:05.775137 (+ 74us) server_negotiation.cc:407] Connection header received
0620 22:03:05.775187 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:05.775188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:05.775207 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:05.775248 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:05.775683 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54370: BlockingRecv error: recv got EOF from 127.0.0.1:54370 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 22:03:06.111503 29546 catalog_manager.cc:1259] Loaded cluster ID: 661a98ade6c247a991f682d45ca99532
I20260620 22:03:06.111605 29546 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881: loading cluster ID for follower catalog manager: success
I20260620 22:03:06.112375 29546 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881: acquiring CA information for follower catalog manager: success
I20260620 22:03:06.112691 29546 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a1fac32e07a4ad08a1a66027d2c9881: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:03:06.185115 29594 catalog_manager.cc:1259] Loaded cluster ID: 661a98ade6c247a991f682d45ca99532
I20260620 22:03:06.185199 29594 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec: loading cluster ID for follower catalog manager: success
I20260620 22:03:06.186014 29594 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec: acquiring CA information for follower catalog manager: success
I20260620 22:03:06.186326 29594 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbd7f1371b0d4dde8f7b6ee2d3c39aec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:03:06.291595 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:06.288879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.6.241.189:38841)
0620 22:03:06.288999 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:06.289001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:06.289066 (+ 65us) server_negotiation.cc:407] Connection header received
0620 22:03:06.289140 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:06.289141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:06.289203 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:06.289314 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:06.289472 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.289970 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.290759 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.290967 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.291047 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:06.291229 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:06.291346 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:06.291435 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:06.291476 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 22:03:06.291495 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":51,"threads_started":1}
I20260620 22:03:06.295807 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:06.293456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.6.241.188:41455)
0620 22:03:06.293559 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:06.293561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:06.293568 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:03:06.293602 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:06.293603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:06.293635 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:06.293707 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:06.293797 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.294441 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.295194 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.295386 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.295439 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:06.295553 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:06.295638 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:06.295689 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:06.295726 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 22:03:06.295741 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
I20260620 22:03:06.300215 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:06.297311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.6.241.189:38841)
0620 22:03:06.297348 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:06.297350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:06.297608 (+ 258us) server_negotiation.cc:407] Connection header received
0620 22:03:06.297683 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:06.297684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:06.297694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:06.297742 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:06.297898 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.298680 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.299440 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.299717 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.299789 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:06.299939 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:06.299970 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:06.300021 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:06.300112 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:03:06.300130 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:03:06.300267 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:06.297366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.6.241.188:41455)
0620 22:03:06.297459 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:06.297461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:06.297698 (+ 237us) server_negotiation.cc:407] Connection header received
0620 22:03:06.297803 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:06.297804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:06.297813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:06.297854 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:06.297993 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.298441 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.299527 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.299821 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.299877 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:06.300008 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:06.300044 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:06.300091 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:06.300163 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 22:03:06.300184 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 22:03:06.300736 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:06.297311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.6.241.190:38719)
0620 22:03:06.297507 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:06.297509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:06.297528 (+ 19us) server_negotiation.cc:407] Connection header received
0620 22:03:06.297568 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:06.297568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:06.297655 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:06.297737 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:06.297861 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.298655 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.299659 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:06.299984 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:06.300072 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:06.300248 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:06.300346 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:06.300592 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:06.300633 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 22:03:06.300655 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":52,"threads_started":1}
I20260620 22:03:06.302403 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 29311
I20260620 22:03:06.307760 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 29386
I20260620 22:03:06.312623 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 29461
2026-06-20T22:03:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2809 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:03:06 dist-test-slave-gl2g krb5kdc[29679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:03:06 dist-test-slave-gl2g krb5kdc[29679](info): set up 2 sockets
Jun 20 22:03:06 dist-test-slave-gl2g krb5kdc[29679](info): commencing operation
krb5kdc: starting...
W20260620 22:03:07.330868 7110 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/test-user.keytab.
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:03:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:03:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.241.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
Entry for principal kudu/127.6.241.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab.
I20260620 22:03:07.441129 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42463
--webserver_interface=127.6.241.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:42463,127.6.241.189:43979,127.6.241.188:33027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:03:07.518285 29692 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:03:07.518482 29692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:03:07.518499 29692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:03:07.518519 29692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:03:07.519632 29692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:03:07.519671 29692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:03:07.519685 29692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:03:07.519698 29692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:03:07.519711 29692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:03:07.521075 29692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:42463,127.6.241.189:43979,127.6.241.188:33027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.190.keytab
--principal=kudu/127.6.241.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.190:42463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.241.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:03:07.521252 29692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:03:07.521405 29692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:03:07.525914 29697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:07.525897 29700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:07.525919 29698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:07.526175 29692 server_base.cc:1034] running on GCE node
I20260620 22:03:07.526374 29692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:03:07.526647 29692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:03:07.527820 29692 hybrid_clock.cc:648] HybridClock initialized: now 1781992987527808 us; error 29 us; skew 500 ppm
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:03:07.530113 29692 init.cc:377] Logged in from keytab as kudu/127.6.241.190@KRBTEST.COM (short username kudu)
I20260620 22:03:07.530517 29692 webserver.cc:460] Webserver started at http://127.6.241.190:46203/ using document root <none> and password file <none>
I20260620 22:03:07.530781 29692 fs_manager.cc:359] Metadata directory not provided
I20260620 22:03:07.530846 29692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:03:07.530994 29692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:03:07.531980 29692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/data/instance:
uuid: "b39daece0b954b2db60e65f687f9b971"
format_stamp: "Formatted at 2026-06-20 22:03:07 on dist-test-slave-gl2g"
server_key: "1e97bc812578f32f0eefb71da406ae0d"
server_key_iv: "fe35065c2da5e560a51cc55c8934bcab"
server_key_version: "encryptionkey@0"
I20260620 22:03:07.532338 29692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/wal/instance:
uuid: "b39daece0b954b2db60e65f687f9b971"
format_stamp: "Formatted at 2026-06-20 22:03:07 on dist-test-slave-gl2g"
server_key: "1e97bc812578f32f0eefb71da406ae0d"
server_key_iv: "fe35065c2da5e560a51cc55c8934bcab"
server_key_version: "encryptionkey@0"
I20260620 22:03:07.533594 29692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:03:07.534260 29707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:07.534423 29692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:07.534524 29692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/wal
uuid: "b39daece0b954b2db60e65f687f9b971"
format_stamp: "Formatted at 2026-06-20 22:03:07 on dist-test-slave-gl2g"
server_key: "1e97bc812578f32f0eefb71da406ae0d"
server_key_iv: "fe35065c2da5e560a51cc55c8934bcab"
server_key_version: "encryptionkey@0"
I20260620 22:03:07.534636 29692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:03:07.542112 29692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:03:07.543859 29692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:03:07.544003 29692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:03:07.547732 29692 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.190:42463
I20260620 22:03:07.547760 29759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.190:42463 every 8 connection(s)
I20260620 22:03:07.548084 29692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/data/info.pb
I20260620 22:03:07.549124 29760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:03:07.550544 29760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:07.555056 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 29692
I20260620 22:03:07.555151 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-0/wal/instance
I20260620 22:03:07.555367 7110 external_mini_cluster.cc:1428] Setting key 34bd96ab0f52d90524c59d378e2c8427
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:03:07.560067 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.551268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42463 (local address 127.0.0.1:59438)
0620 22:03:07.551435 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.551470 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:07.551881 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:07.552116 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:07.552119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:07.552359 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:07.552505 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.552508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.553446 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.553447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.554081 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.554082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.554506 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.554507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.554556 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.555092 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:07.555099 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:07.556836 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:07.558620 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.558626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.558628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.558829 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.559077 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.559077 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.559078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.559163 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.559495 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:07.559500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:07.559556 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 22:03:07.559659 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 22:03:07.559877 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":94,"thread_start_us":69,"threads_started":1}
I20260620 22:03:07.560070 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.551435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.6.241.190:42463)
0620 22:03:07.551544 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.551548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.551585 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:03:07.552001 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.552003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.552018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.552092 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.552582 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.553371 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.554147 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.554422 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.554497 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.556941 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.556948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.556949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.556955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.558497 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.558923 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.558925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.558925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.559002 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.559226 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.559227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.559228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.559412 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.559514 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.559696 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.559817 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
I20260620 22:03:07.560775 29760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:07.561102 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.560980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:43979 (local address 127.0.0.1:55162)
0620 22:03:07.561026 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.561077 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 22:03:07.561221 29711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
I20260620 22:03:07.561615 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.561526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:43979 (local address 127.0.0.1:55164)
0620 22:03:07.561573 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.561595 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:03:07.561731 29760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.189:43979: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.6.241.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab.
Entry for principal kudu/127.6.241.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab.
I20260620 22:03:07.577337 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.189:43979
--webserver_interface=127.6.241.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab
--principal=kudu/127.6.241.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:42463,127.6.241.189:43979,127.6.241.188:33027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:03:07.610919 29760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } attempt: 1
I20260620 22:03:07.611389 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.611232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:43979 (local address 127.0.0.1:55180)
0620 22:03:07.611299 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.611347 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:03:07.611769 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.611676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:43979 (local address 127.0.0.1:55192)
0620 22:03:07.611721 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.611749 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 22:03:07.611881 29760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.189:43979: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
W20260620 22:03:07.657253 29767 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:03:07.657451 29767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:03:07.657469 29767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:03:07.657486 29767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:03:07.658579 29767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:03:07.658632 29767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:03:07.658646 29767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:03:07.658659 29767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:03:07.658671 29767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:03:07.660154 29767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:42463,127.6.241.189:43979,127.6.241.188:33027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.189.keytab
--principal=kudu/127.6.241.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.189:43979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.241.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:03:07.660341 29767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:03:07.660494 29767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:03:07.665126 29772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:07.665155 29775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:07.665179 29767 server_base.cc:1034] running on GCE node
W20260620 22:03:07.665136 29773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:07.665645 29767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:03:07.665928 29767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:03:07.667109 29767 hybrid_clock.cc:648] HybridClock initialized: now 1781992987667096 us; error 27 us; skew 500 ppm
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:03:07.669445 29767 init.cc:377] Logged in from keytab as kudu/127.6.241.189@KRBTEST.COM (short username kudu)
I20260620 22:03:07.669905 29767 webserver.cc:460] Webserver started at http://127.6.241.189:44817/ using document root <none> and password file <none>
I20260620 22:03:07.670141 29767 fs_manager.cc:359] Metadata directory not provided
I20260620 22:03:07.670208 29767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:03:07.670361 29767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:03:07.671484 29767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/data/instance:
uuid: "ac4669e1582a406bb32fdad82206ef37"
format_stamp: "Formatted at 2026-06-20 22:03:07 on dist-test-slave-gl2g"
server_key: "b16162f947f7d7683590cb81783caf62"
server_key_iv: "f3f20164979741fb6ee616200834beac"
server_key_version: "encryptionkey@0"
I20260620 22:03:07.671855 29767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/wal/instance:
uuid: "ac4669e1582a406bb32fdad82206ef37"
format_stamp: "Formatted at 2026-06-20 22:03:07 on dist-test-slave-gl2g"
server_key: "b16162f947f7d7683590cb81783caf62"
server_key_iv: "f3f20164979741fb6ee616200834beac"
server_key_version: "encryptionkey@0"
I20260620 22:03:07.673131 29767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:03:07.673825 29782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:07.673980 29767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:03:07.674078 29767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/wal
uuid: "ac4669e1582a406bb32fdad82206ef37"
format_stamp: "Formatted at 2026-06-20 22:03:07 on dist-test-slave-gl2g"
server_key: "b16162f947f7d7683590cb81783caf62"
server_key_iv: "f3f20164979741fb6ee616200834beac"
server_key_version: "encryptionkey@0"
I20260620 22:03:07.674154 29767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:03:07.680014 29760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } attempt: 2
I20260620 22:03:07.680394 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.680275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:43979 (local address 127.0.0.1:55196)
0620 22:03:07.680331 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.680367 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 22:03:07.680693 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.680603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:43979 (local address 127.0.0.1:55200)
0620 22:03:07.680647 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.680669 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:03:07.680776 29760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.189:43979: Network error: Client connection negotiation failed: client connection to 127.6.241.189:43979: connect: Connection refused (error 111)
I20260620 22:03:07.700811 29767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:03:07.702890 29767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:03:07.703049 29767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:03:07.706923 29767 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.189:43979
I20260620 22:03:07.706976 29834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.189:43979 every 8 connection(s)
I20260620 22:03:07.707350 29767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/data/info.pb
I20260620 22:03:07.708343 29835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:03:07.709879 29835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:07.711707 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 29767
I20260620 22:03:07.711817 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-1/wal/instance
I20260620 22:03:07.712023 7110 external_mini_cluster.cc:1428] Setting key 9b4b48d36dddfd421fbae1ab52168548
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:03:07.719715 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.710549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42463 (local address 127.0.0.1:59440)
0620 22:03:07.710731 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.710767 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:07.711248 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:07.711510 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:07.711529 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:07.711905 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:07.712200 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.712206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.712884 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.712885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.713899 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.713901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.714324 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.714326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.714480 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.714962 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:07.714980 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:07.716543 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:07.718385 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.718392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.718393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.718651 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.718876 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.718876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.718876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.718972 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.719265 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:07.719269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:07.719374 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 22:03:07.719444 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:03:07.719556 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260620 22:03:07.719719 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.710578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.6.241.190:42463)
0620 22:03:07.710660 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.710662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.710852 (+ 190us) server_negotiation.cc:407] Connection header received
0620 22:03:07.711370 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.711371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.711387 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.711437 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.712306 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.712784 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.713996 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.714218 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.714304 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.716641 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.716643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.716643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.716652 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.718296 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.718733 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.718734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.718735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.718799 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.719048 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.719049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.719049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.719192 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.719263 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.719514 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.719610 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.6.241.188@KRBTEST.COM; defaulting to no policy
I20260620 22:03:07.720377 29835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.241.188@KRBTEST.COM" created.
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
I20260620 22:03:07.727748 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.720654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:43979 (local address 127.0.0.1:55212)
0620 22:03:07.720708 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.720720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:07.720764 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:07.721014 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:07.721015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:07.721118 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:07.721165 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.721166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.722070 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.722071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.722548 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.722549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.723172 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.723173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.723263 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.723641 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:07.723646 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:07.724747 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:07.726795 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.726796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.726796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.726944 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.727174 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.727175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.727175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.727222 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.727546 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:07.727546 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:07.727577 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 22:03:07.727623 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:03:07.727689 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 22:03:07.728001 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.720735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.6.241.189:43979)
0620 22:03:07.720885 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.720889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.720905 (+ 16us) server_negotiation.cc:407] Connection header received
0620 22:03:07.720924 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.720925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.720935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.720999 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.721229 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.721996 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.722656 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.723097 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.723153 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.724826 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.724835 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.724835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.724842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.726719 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.727028 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.727031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.727032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.727099 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.727287 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.727288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.727288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.727470 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.727558 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.727662 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.727756 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260620 22:03:07.728578 29835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:07.728878 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.728767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:34974)
0620 22:03:07.728802 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.728852 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 22:03:07.729040 29785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
I20260620 22:03:07.729468 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.729358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:34978)
0620 22:03:07.729408 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.729438 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 22:03:07.729620 29835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:33027: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.241.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab.
Entry for principal kudu/127.6.241.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab.
I20260620 22:03:07.737625 7110 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dSvQ8/build/release/bin/kudu
/tmp/dist-test-task4dSvQ8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.188:33027
--webserver_interface=127.6.241.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab
--principal=kudu/127.6.241.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.241.148:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.241.190:42463,127.6.241.189:43979,127.6.241.188:33027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:03:07.771911 29760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } attempt: 3
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
I20260620 22:03:07.778818 29835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } attempt: 1
I20260620 22:03:07.779093 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.772278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:43979 (local address 127.0.0.1:55216)
0620 22:03:07.772320 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.772333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:07.772383 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:07.772573 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:07.772574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:07.772737 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:07.772799 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.772802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.773459 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.773460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.773985 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.773987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.774392 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.774393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.774474 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.774962 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:07.774967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:07.776291 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:07.777988 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.777989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.777990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.778161 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.778366 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.778366 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.778367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.778431 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.778758 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:07.778759 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:07.778804 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 22:03:07.778892 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:03:07.778988 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 22:03:07.779189 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.779080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:34982)
0620 22:03:07.779129 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.779158 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 22:03:07.779477 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.772381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.6.241.189:43979)
0620 22:03:07.772454 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.772456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.772463 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:03:07.772494 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.772494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.772504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.772566 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.772890 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.773374 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.774079 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.774314 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.774395 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.776380 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.776382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.776383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.776429 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.777901 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.778243 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.778244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.778244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.778303 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.778495 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.778496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.778497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.778674 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.778765 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.778917 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.779054 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:03:07.779520 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.779431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:34986)
0620 22:03:07.779473 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.779500 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:03:07.779788 29760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } has no permanent_uuid. Determining permanent_uuid...
W20260620 22:03:07.779826 29835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:33027: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
I20260620 22:03:07.780099 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.780015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:34998)
0620 22:03:07.780055 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.780082 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 22:03:07.780408 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.780321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:35010)
0620 22:03:07.780367 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.780389 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:03:07.780499 29760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:33027: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
I20260620 22:03:07.811617 29760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } attempt: 1
I20260620 22:03:07.812043 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.811929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:35014)
0620 22:03:07.811981 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.812017 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:03:07.812376 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.812286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:35022)
0620 22:03:07.812328 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.812350 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 22:03:07.812461 29760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:33027: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
W20260620 22:03:07.817116 29842 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:03:07.817332 29842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:03:07.817374 29842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:03:07.817410 29842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:03:07.818531 29842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:03:07.818598 29842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:03:07.818646 29842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:03:07.818677 29842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:03:07.818703 29842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:03:07.820175 29842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.241.148:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.241.190:42463,127.6.241.189:43979,127.6.241.188:33027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/krb5kdc/kudu_127.6.241.188.keytab
--principal=kudu/127.6.241.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.241.188:33027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.241.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 20:55:05 UTC on ccycloud.quasar-ypstkz.root.comops.site
build id 4457724
I20260620 22:03:07.820446 29842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:03:07.820658 29842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:03:07.825127 29847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:07.825141 29848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:03:07.825145 29850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:03:07.825493 29842 server_base.cc:1034] running on GCE node
I20260620 22:03:07.825682 29842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:03:07.825891 29842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:03:07.827049 29842 hybrid_clock.cc:648] HybridClock initialized: now 1781992987827034 us; error 26 us; skew 500 ppm
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:03:07.829294 29842 init.cc:377] Logged in from keytab as kudu/127.6.241.188@KRBTEST.COM (short username kudu)
I20260620 22:03:07.829720 29842 webserver.cc:460] Webserver started at http://127.6.241.188:36911/ using document root <none> and password file <none>
I20260620 22:03:07.829932 29842 fs_manager.cc:359] Metadata directory not provided
I20260620 22:03:07.830003 29842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:03:07.830153 29842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:03:07.831228 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/data/instance:
uuid: "ece03175bdc14588a40a73c2e309b857"
format_stamp: "Formatted at 2026-06-20 22:03:07 on dist-test-slave-gl2g"
server_key: "a6abd0463dda9af0106370c08cbdc532"
server_key_iv: "24f7f026aad1ad11cd21d26d361a3534"
server_key_version: "encryptionkey@0"
I20260620 22:03:07.831574 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/wal/instance:
uuid: "ece03175bdc14588a40a73c2e309b857"
format_stamp: "Formatted at 2026-06-20 22:03:07 on dist-test-slave-gl2g"
server_key: "a6abd0463dda9af0106370c08cbdc532"
server_key_iv: "24f7f026aad1ad11cd21d26d361a3534"
server_key_version: "encryptionkey@0"
I20260620 22:03:07.832928 29842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:03:07.833629 29857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:03:07.833794 29842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:03:07.833889 29842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/data,/tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/wal
uuid: "ece03175bdc14588a40a73c2e309b857"
format_stamp: "Formatted at 2026-06-20 22:03:07 on dist-test-slave-gl2g"
server_key: "a6abd0463dda9af0106370c08cbdc532"
server_key_iv: "24f7f026aad1ad11cd21d26d361a3534"
server_key_version: "encryptionkey@0"
I20260620 22:03:07.833967 29842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:03:07.847962 29835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } attempt: 2
I20260620 22:03:07.848341 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.848218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:35028)
0620 22:03:07.848287 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.848316 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 22:03:07.848664 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.848571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:35032)
0620 22:03:07.848615 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.848639 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:03:07.848742 29835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.241.188:33027: Network error: Client connection negotiation failed: client connection to 127.6.241.188:33027: connect: Connection refused (error 111)
I20260620 22:03:07.850492 29842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:03:07.852298 29842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:03:07.852401 29842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:03:07.856254 29842 rpc_server.cc:307] RPC server started. Bound to: 127.6.241.188:33027
I20260620 22:03:07.856285 29909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.241.188:33027 every 8 connection(s)
I20260620 22:03:07.856595 29842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/data/info.pb
I20260620 22:03:07.857457 29910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:03:07.858997 29910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:07.861682 7110 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dSvQ8/build/release/bin/kudu as pid 29842
I20260620 22:03:07.861791 7110 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dSvQ8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992718478901-7110-0/minicluster-data/master-2/wal/instance
I20260620 22:03:07.862005 7110 external_mini_cluster.cc:1428] Setting key 8c81fa6c17f0b0da3a495aeaa697ef18
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:03:07.867882 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.859615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.6.241.190:42463)
0620 22:03:07.859709 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.859712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.859843 (+ 131us) server_negotiation.cc:407] Connection header received
0620 22:03:07.860255 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.860256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.860266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.860316 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.860827 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.861279 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.862365 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.862691 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.862769 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.864891 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.864893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.864893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.864902 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.866405 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.866892 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.866893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.866894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.866968 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.867234 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.867235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.867235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.867371 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.867440 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.867687 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.867768 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 22:03:07.867959 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.859569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.190:42463 (local address 127.0.0.1:59452)
0620 22:03:07.859735 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.859766 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:07.860180 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:07.860363 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:07.860366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:07.860587 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:07.860759 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.860763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.861357 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.861358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.862289 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.862292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.862795 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.862797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.862939 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.863431 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:07.863439 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:07.864802 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:07.866527 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.866534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.866536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.866803 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.867055 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.867055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.867056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.867163 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.867437 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:07.867441 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:07.867538 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 22:03:07.867664 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 22:03:07.867789 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260620 22:03:07.868561 29910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:03:07.871037 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.862372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.6.241.188:33027)
0620 22:03:07.862541 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.862545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.862558 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:03:07.862644 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.862647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.862665 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.862738 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.863803 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.864581 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.865336 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.865624 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.865705 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.868102 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.868109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.868109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.868116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.869788 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.870171 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.870173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.870174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.870235 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.870439 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.870440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.870440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.870575 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.870756 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.870844 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.870920 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
I20260620 22:03:07.875253 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.868855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.6.241.189:43979)
0620 22:03:07.868898 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.868899 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.868908 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:03:07.868970 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.868971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.868979 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.869032 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.869296 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.869723 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.870442 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.870729 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.870813 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.872794 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.872795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.872795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.872803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.874234 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.874529 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.874531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.874531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.874615 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.874817 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.874818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.874818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.874919 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.874973 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.875140 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.875197 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:03:07.875535 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.868798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.189:43979 (local address 127.0.0.1:55230)
0620 22:03:07.868855 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.868866 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:07.868908 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:07.869035 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:07.869036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:07.869169 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:07.869210 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.869211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.869800 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.869801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.870363 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.870365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.870813 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.870814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.870883 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.871430 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:07.871433 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:07.872723 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:07.874315 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.874316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.874317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.874477 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.874678 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.874679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.874679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.874748 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.874988 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:07.874989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:07.875034 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 22:03:07.875121 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 22:03:07.875212 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 22:03:07.875953 29910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.190@KRBTEST.COM
I20260620 22:03:07.879307 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.871703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.6.241.190:42463)
0620 22:03:07.871735 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.871737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.871786 (+ 49us) server_negotiation.cc:407] Connection header received
0620 22:03:07.871850 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.871851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.871859 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.871903 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.872458 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.873038 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.873901 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.874216 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.874293 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.876768 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.876769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.876770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.876777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.878168 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.878486 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.878487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.878487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.878550 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.878794 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.878795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.878795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.878928 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.879079 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.879164 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.879222 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.188@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:03:07.882863 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.876195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:35050)
0620 22:03:07.876276 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.876286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:07.876338 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:07.876452 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:07.876453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:07.876611 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:07.876645 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.876646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.877558 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.877559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.878171 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.878173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.878571 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.878572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.878663 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.879057 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:07.879061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:07.880278 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:07.881796 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.881797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.881798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.882005 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.882183 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.882184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.882184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.882245 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.882490 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:07.882491 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:07.882540 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 22:03:07.882627 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 22:03:07.882741 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 22:03:07.882977 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.876239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.6.241.188:33027)
0620 22:03:07.876276 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.876278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.876324 (+ 46us) server_negotiation.cc:407] Connection header received
0620 22:03:07.876384 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.876385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.876393 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.876439 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.876741 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.877469 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.878254 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.878498 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.878566 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.880356 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.880358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.880358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.880365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.881722 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.882072 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.882073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.882073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.882116 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.882316 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.882316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.882317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.882428 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.882493 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.882666 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.882820 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":22}
I20260620 22:03:07.884230 29910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857: Bootstrap starting.
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.241.189@KRBTEST.COM
I20260620 22:03:07.885241 29910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857: Neither blocks nor log segments found. Creating new log.
I20260620 22:03:07.885561 29910 log.cc:824] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857: Log is configured to *not* fsync() on all Append() calls
I20260620 22:03:07.886320 29910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857: No bootstrap required, opened a new log
I20260620 22:03:07.887609 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.880379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.6.241.189:43979)
0620 22:03:07.880440 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.880442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.880451 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:03:07.880471 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.880471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.880480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.880527 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.880968 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.881403 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.882208 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.882491 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.882548 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.885045 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.885046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.885046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.885051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.886457 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.886863 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.886864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.886865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.886921 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.887112 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.887112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.887113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.887225 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.887397 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.887431 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.887500 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:03:07.887593 29760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } attempt: 2
I20260620 22:03:07.888118 29910 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } }
I20260620 22:03:07.888269 29910 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece03175bdc14588a40a73c2e309b857, State: Initialized, Role: FOLLOWER
I20260620 22:03:07.888360 29910 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } }
I20260620 22:03:07.888681 29910 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:03:07.889034 7110 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:03:07.890122 29916 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:07.890467 29916 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:03:07.890503 29927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:03:07.890548 29927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:03:07.889947 29734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b39daece0b954b2db60e65f687f9b971"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59462
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.190@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:03:07.891341 29842 master_runner.cc:186] Error getting master registration for 127.6.241.190:42463: OK, instance_id { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" instance_seqno: 1781992987546686 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:03:07.894205 29842 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:03:07.895479 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.887842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:35064)
0620 22:03:07.887872 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.887883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:07.887919 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:07.888048 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:07.888049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:07.888187 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:07.888241 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.888243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.888935 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.888936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.890097 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.890100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.890708 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.890710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.890764 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.891297 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:07.891302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:07.892618 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:07.894386 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.894387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.894387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.894618 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.894815 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.894816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.894816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.894875 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.895170 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:07.895170 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:07.895209 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 22:03:07.895282 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:03:07.895371 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260620 22:03:07.895485 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.887924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.6.241.188:33027)
0620 22:03:07.887964 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.887966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.887973 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:03:07.887989 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.887990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.887999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.888046 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.888321 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.888856 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.890286 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.890577 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.892714 (+ 2137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.892755 (+ 41us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.892756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.892756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.892760 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.894299 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.894692 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.894693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.894693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.894750 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.894944 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.894945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.894945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.895098 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.895171 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.895315 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.895390 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:03:07.897042 29760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971: Bootstrap starting.
I20260620 22:03:07.897751 29760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971: Neither blocks nor log segments found. Creating new log.
I20260620 22:03:07.898038 29760 log.cc:824] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971: Log is configured to *not* fsync() on all Append() calls
I20260620 22:03:07.898900 29760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971: No bootstrap required, opened a new log
I20260620 22:03:07.900200 29760 raft_consensus.cc:348] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } }
I20260620 22:03:07.900332 29760 raft_consensus.cc:729] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39daece0b954b2db60e65f687f9b971, State: Initialized, Role: FOLLOWER
I20260620 22:03:07.900415 29760 consensus_queue.cc:260] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } }
I20260620 22:03:07.900719 29760 sys_catalog.cc:564] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:03:07.900702 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:07.900758 29734 raft_consensus.cc:482] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 22:03:07.900848 29734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:03:07.901561 29734 raft_consensus.cc:504] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } }
I20260620 22:03:07.901841 29734 leader_election.cc:288] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [CANDIDATE]: Term 1 election: Requested vote from peers ac4669e1582a406bb32fdad82206ef37 (127.6.241.189:43979), ece03175bdc14588a40a73c2e309b857 (127.6.241.188:33027)
I20260620 22:03:07.902384 29884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b39daece0b954b2db60e65f687f9b971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ece03175bdc14588a40a73c2e309b857"
I20260620 22:03:07.902503 29884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:03:07.903471 29884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b39daece0b954b2db60e65f687f9b971 in term 1.
W20260620 22:03:07.903537 29941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:03:07.903581 29941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:03:07.904002 29710 leader_election.cc:302] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b39daece0b954b2db60e65f687f9b971, ece03175bdc14588a40a73c2e309b857; no voters:
I20260620 22:03:07.904196 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:07.904237 29947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:03:07.904539 29947 raft_consensus.cc:686] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [term 1 LEADER]: Becoming Leader. State: Replica: b39daece0b954b2db60e65f687f9b971, State: Running, Role: LEADER
I20260620 22:03:07.904747 29947 consensus_queue.cc:237] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } }
I20260620 22:03:07.905632 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b39daece0b954b2db60e65f687f9b971. Latest consensus state: current_term: 1 leader_uuid: "b39daece0b954b2db60e65f687f9b971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:07.905735 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:07.903745 29692 master_runner.cc:186] Error getting master registration for 127.6.241.189:43979: OK, instance_id { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" instance_seqno: 1781992987705829 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:03:07.905854 29692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:03:07.910781 29951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:03:07.910948 29951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:03:07.912632 29884 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [term 1 FOLLOWER]: Refusing update from remote peer b39daece0b954b2db60e65f687f9b971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:03:07.913125 29930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:03:07.914718 29916 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b39daece0b954b2db60e65f687f9b971. Latest consensus state: current_term: 1 leader_uuid: "b39daece0b954b2db60e65f687f9b971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:07.914827 29916 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:07.918700 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.904476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.6.241.188:33027)
0620 22:03:07.905940 (+ 1464us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.905942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.910763 (+ 4821us) server_negotiation.cc:407] Connection header received
0620 22:03:07.910821 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.910823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.910845 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.910893 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.911720 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.912398 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.913694 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.914045 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.914095 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.915755 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.915756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.915756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.915763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.917407 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.917792 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.917793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.917793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.917850 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.918038 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.918039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.918040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.918176 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.918277 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.918469 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.918557 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1444}
I20260620 22:03:07.920715 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b39daece0b954b2db60e65f687f9b971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:07.920801 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:07.921128 29953 mvcc.cc:205] Tried to move back new op lower bound from 7299043278487412736 to 7299043278459424768. Current Snapshot: MvccSnapshot[applied={T|T < 7299043278487412736}]
I20260620 22:03:07.921277 29916 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b39daece0b954b2db60e65f687f9b971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:07.921366 29916 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:07.921525 29954 mvcc.cc:205] Tried to move back new op lower bound from 7299043278487412736 to 7299043278459424768. Current Snapshot: MvccSnapshot[applied={T|T < 7299043278487412736}]
I20260620 22:03:07.926973 29951 catalog_manager.cc:1347] Generated new cluster ID: a3425bca072f4cae9891e9009d65f1fe
I20260620 22:03:07.927024 29951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:03:07.928388 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.905081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.6.241.190:42463)
0620 22:03:07.905116 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.905118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.913091 (+ 7973us) server_negotiation.cc:407] Connection header received
0620 22:03:07.915722 (+ 2631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.915724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.915750 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.915810 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.919629 (+ 3819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.920308 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.921243 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.921536 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.921575 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.923633 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.923634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.923634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.923640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.926114 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.927294 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.927295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.927295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.927363 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.927656 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.927657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.927671 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.927809 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.927902 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.928222 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.928300 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:03:07.928491 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.906831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.6.241.189:43979)
0620 22:03:07.906893 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.906895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.918719 (+ 11824us) server_negotiation.cc:407] Connection header received
0620 22:03:07.918834 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.918836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.918847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.918876 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.919507 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.920232 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.923693 (+ 3461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.924008 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.924050 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.925458 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.925459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.925459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.925466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.927045 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.927681 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.927682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.927682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.927729 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.927927 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.927928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.927928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.928049 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.928115 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.928316 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.928383 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:03:07.939846 29835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } attempt: 3
Jun 20 22:03:07 dist-test-slave-gl2g krb5kdc[29679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992987, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.241.189@KRBTEST.COM for kudu/127.6.241.188@KRBTEST.COM
I20260620 22:03:07.947582 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.940085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.241.188:33027 (local address 127.0.0.1:35082)
0620 22:03:07.940128 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 22:03:07.940138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:03:07.940174 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:03:07.940326 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:03:07.940327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:03:07.940492 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:03:07.940526 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.940527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.941423 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.941424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.941969 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:03:07.941971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.942753 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.942754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:03:07.942816 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.943224 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:03:07.943229 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:03:07.944661 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:03:07.946540 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.946541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.946541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.946744 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.946967 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:03:07.946967 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:03:07.946968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:03:07.947028 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:03:07.947267 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:03:07.947267 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:03:07.947310 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:03:07.947395 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:03:07.947489 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 22:03:07.947580 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:07.940130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.6.241.188:33027)
0620 22:03:07.940165 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:07.940168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:07.940176 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:03:07.940242 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:07.940242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:07.940252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:07.940309 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:07.940617 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.941342 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.942361 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:07.942693 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:07.942753 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:07.944734 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:03:07.944735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:03:07.944735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:03:07.944743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:03:07.946472 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.946838 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.946839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.946840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.946896 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:03:07.947091 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:03:07.947092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:03:07.947092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:03:07.947210 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:03:07.947268 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:07.947418 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 22:03:07.947489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:03:07.949211 29835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37: Bootstrap starting.
I20260620 22:03:07.949942 29835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37: Neither blocks nor log segments found. Creating new log.
I20260620 22:03:07.950239 29835 log.cc:824] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37: Log is configured to *not* fsync() on all Append() calls
I20260620 22:03:07.951653 29835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37: No bootstrap required, opened a new log
I20260620 22:03:07.953567 29835 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } }
I20260620 22:03:07.953722 29835 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4669e1582a406bb32fdad82206ef37, State: Initialized, Role: FOLLOWER
I20260620 22:03:07.953807 29835 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } }
I20260620 22:03:07.954114 29835 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:03:07.954154 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b39daece0b954b2db60e65f687f9b971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac4669e1582a406bb32fdad82206ef37"
I20260620 22:03:07.954246 29809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:03:07.955091 29809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b39daece0b954b2db60e65f687f9b971 in term 1.
I20260620 22:03:07.955533 29808 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [term 1 FOLLOWER]: Refusing update from remote peer b39daece0b954b2db60e65f687f9b971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:03:07.955874 29947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:03:07.956636 29970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:03:07.956687 29970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:03:07.954082 29959 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:07.959208 29959 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:07.959306 29959 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b39daece0b954b2db60e65f687f9b971. Latest consensus state: current_term: 1 leader_uuid: "b39daece0b954b2db60e65f687f9b971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:07.959419 29959 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:07.959497 29959 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b39daece0b954b2db60e65f687f9b971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:07.959581 29959 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:03:07.960019 29973 mvcc.cc:205] Tried to move back new op lower bound from 7299043278487412736 to 7299043278459424768. Current Snapshot: MvccSnapshot[applied={T|T < 7299043278487412736}]
I20260620 22:03:08.000113 29947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b39daece0b954b2db60e65f687f9b971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b39daece0b954b2db60e65f687f9b971" member_type: VOTER last_known_addr { host: "127.6.241.190" port: 42463 } } peers { permanent_uuid: "ac4669e1582a406bb32fdad82206ef37" member_type: VOTER last_known_addr { host: "127.6.241.189" port: 43979 } } peers { permanent_uuid: "ece03175bdc14588a40a73c2e309b857" member_type: VOTER last_known_addr { host: "127.6.241.188" port: 33027 } } }
I20260620 22:03:08.000237 29947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971 [sys.catalog]: This master's current role is: LEADER
I20260620 22:03:08.001421 29951 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:03:08.002274 29951 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:03:08.010968 29951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b39daece0b954b2db60e65f687f9b971: Generated new TSK 0
I20260620 22:03:08.011125 29951 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:03:08.038743 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.036839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.6.241.189:43979)
0620 22:03:08.036882 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.036884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.038009 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 22:03:08.038168 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.038169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.038182 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.038215 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.038670 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55256: BlockingRecv error: recv got EOF from 127.0.0.1:55256 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 22:03:08.039495 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.037258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.6.241.188:33027)
0620 22:03:08.037299 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.037301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.038976 (+ 1675us) server_negotiation.cc:407] Connection header received
0620 22:03:08.039073 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.039074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.039084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.039113 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.039434 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35096: BlockingRecv error: recv got EOF from 127.0.0.1:35096 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 22:03:08.040198 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.037671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.6.241.190:42463)
0620 22:03:08.037727 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.037728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.039707 (+ 1979us) server_negotiation.cc:407] Connection header received
0620 22:03:08.039797 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.039798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.039808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.039837 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.040127 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59482: BlockingRecv error: recv got EOF from 127.0.0.1:59482 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 22:03:08.042438 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.041729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.6.241.189:43979)
0620 22:03:08.041762 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.041763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.041947 (+ 184us) server_negotiation.cc:407] Connection header received
0620 22:03:08.042046 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.042046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.042062 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.042090 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.042395 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55258: BlockingRecv error: recv got EOF from 127.0.0.1:55258 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:03:08.045469 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.044678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.6.241.189:43979)
0620 22:03:08.044710 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.044712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.044966 (+ 254us) server_negotiation.cc:407] Connection header received
0620 22:03:08.045057 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.045058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.045068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.045096 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.045420 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55260: BlockingRecv error: recv got EOF from 127.0.0.1:55260 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 22:03:08.049340 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.048633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.6.241.189:43979)
0620 22:03:08.048663 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.048665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.048865 (+ 200us) server_negotiation.cc:407] Connection header received
0620 22:03:08.048955 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.048956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.048966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.048994 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.049298 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55262: BlockingRecv error: recv got EOF from 127.0.0.1:55262 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 22:03:08.055264 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.054484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.6.241.189:43979)
0620 22:03:08.054517 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.054519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.054761 (+ 242us) server_negotiation.cc:407] Connection header received
0620 22:03:08.054858 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.054859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.054870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.054898 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.055222 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55278: BlockingRecv error: recv got EOF from 127.0.0.1:55278 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 22:03:08.065940 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.064772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.6.241.189:43979)
0620 22:03:08.064812 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.064815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.065226 (+ 411us) server_negotiation.cc:407] Connection header received
0620 22:03:08.065408 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.065409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.065422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.065455 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.065872 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55292: BlockingRecv error: recv got EOF from 127.0.0.1:55292 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:03:08.084059 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.083196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.6.241.189:43979)
0620 22:03:08.083238 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.083240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.083500 (+ 260us) server_negotiation.cc:407] Connection header received
0620 22:03:08.083596 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.083598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.083608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.083638 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.084003 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55300: BlockingRecv error: recv got EOF from 127.0.0.1:55300 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:03:08.118093 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.117339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.6.241.189:43979)
0620 22:03:08.117397 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.117399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.117562 (+ 163us) server_negotiation.cc:407] Connection header received
0620 22:03:08.117599 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.117600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.117614 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.117701 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.118033 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55304: BlockingRecv error: recv got EOF from 127.0.0.1:55304 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 22:03:08.183966 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.183294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.6.241.189:43979)
0620 22:03:08.183410 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.183412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.183465 (+ 53us) server_negotiation.cc:407] Connection header received
0620 22:03:08.183533 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.183533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.183543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.183588 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.183897 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55308: BlockingRecv error: recv got EOF from 127.0.0.1:55308 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 22:03:08.313872 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.313195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.6.241.189:43979)
0620 22:03:08.313272 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.313274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.313329 (+ 55us) server_negotiation.cc:407] Connection header received
0620 22:03:08.313371 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.313372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.313382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.313433 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.313775 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55320: BlockingRecv error: recv got EOF from 127.0.0.1:55320 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 22:03:08.571835 29839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:03:08.571233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.6.241.189:43979)
0620 22:03:08.571332 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:08.571333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:08.571343 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:03:08.571369 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:08.571370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:08.571380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:08.571450 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:03:08.571758 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55324: BlockingRecv error: recv got EOF from 127.0.0.1:55324 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 22:03:08.890954 29927 catalog_manager.cc:1259] Loaded cluster ID: a3425bca072f4cae9891e9009d65f1fe
I20260620 22:03:08.891052 29927 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857: loading cluster ID for follower catalog manager: success
I20260620 22:03:08.891841 29927 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857: acquiring CA information for follower catalog manager: success
I20260620 22:03:08.892134 29927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ece03175bdc14588a40a73c2e309b857: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:03:08.957087 29970 catalog_manager.cc:1259] Loaded cluster ID: a3425bca072f4cae9891e9009d65f1fe
I20260620 22:03:08.957163 29970 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37: loading cluster ID for follower catalog manager: success
I20260620 22:03:08.957955 29970 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37: acquiring CA information for follower catalog manager: success
I20260620 22:03:08.958202 29970 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac4669e1582a406bb32fdad82206ef37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:03:09.087563 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:09.085059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.6.241.189:43979)
0620 22:03:09.085233 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:09.085235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:09.085273 (+ 38us) server_negotiation.cc:407] Connection header received
0620 22:03:09.085346 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:09.085346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:09.085397 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:09.085473 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:09.085612 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.086112 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.086860 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.087049 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.087099 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:09.087224 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:09.087330 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:09.087387 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:09.087419 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 22:03:09.087436 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260620 22:03:09.091732 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:09.089454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.6.241.188:33027)
0620 22:03:09.089550 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:09.089553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:09.089583 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:03:09.089648 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:09.089649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:09.089690 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:09.089799 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:09.089863 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.090328 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.091105 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.091304 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.091360 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:09.091467 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:09.091562 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:09.091613 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:09.091642 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 22:03:09.091658 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":50,"threads_started":1}
I20260620 22:03:09.096290 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:09.093370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.6.241.190:42463)
0620 22:03:09.093566 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:09.093568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:09.093822 (+ 254us) server_negotiation.cc:407] Connection header received
0620 22:03:09.093872 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:09.093873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:09.093926 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:09.094007 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:09.094100 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.094584 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.095478 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.095684 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.095772 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:09.095959 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:09.096057 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:09.096123 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:09.096182 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 22:03:09.096207 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":47,"threads_started":1}
I20260620 22:03:09.096624 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:09.093363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.6.241.189:43979)
0620 22:03:09.093561 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:09.093563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:09.093684 (+ 121us) server_negotiation.cc:407] Connection header received
0620 22:03:09.093733 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:09.093734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:09.093745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:09.093798 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:09.093934 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.094686 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.095863 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.096077 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.096154 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:09.096339 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:09.096369 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:09.096401 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:09.096512 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 22:03:09.096532 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260620 22:03:09.096748 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:03:09.093505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.6.241.188:33027)
0620 22:03:09.093597 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 22:03:09.093598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:03:09.093608 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:03:09.093649 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:03:09.093650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:03:09.093687 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:03:09.093745 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:03:09.093890 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.094646 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.095842 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:03:09.096107 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:03:09.096162 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:03:09.096344 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:03:09.096378 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:03:09.096423 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:03:09.096590 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:03:09.096612 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 22:03:09.102171 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 29692
I20260620 22:03:09.107612 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 29767
I20260620 22:03:09.112351 7110 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dSvQ8/build/release/bin/kudu with pid 29842
2026-06-20T22:03:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2799 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5609 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (270640 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST